To pool or not to pool, that is the question: Change to Anova RM post-hoc tests in version 2.7.6

[Mik]

I noticed the change in the way post hoc test are carried in v.2.7.6 ("2.7.6 – 25 August 2025, Fix to Post-hoc in RM ANOVA,
https://www.jamovi.org/release-notes.html). Now jamovi assumes that variances are unequal and therefore, the standard error is not pooled anymore. I question this change, because the default assumption of the RM ANOVA test is that variances are equal. This was also explained here: viewtopic.php?t=114

The repeated measure ANOVA can of course be adjusted when the sphericity test is significant. This should however not be the default behaviour of the procedure to compute post hoc tests with unequal standard error.

Would it be possible to have an option to check where the user could ask jamovi to use non-pooled error terms for the post hoc tests ? I think that would be a better choice, because the use of non-pooled error terms might conflict with the default assumption of the RM ANOVA. That idea is also supported here: https://github.com/jamovi/jamovi/issues/660

[jonathon]

Hi,

The post-hocs in RM ANOVA were fixed in the 2.7.6, but this did not change the way they were calculated.

If you're getting different numbers between 2.6 and 2.7, then this is something we need to investigate. could you confirm that you're getting different numbers between the 2.6 and the 2.7?

with thanks

jonathon

[Mik]

Hi,

I do not know exactly when this change was implemented in jamovi because I was using an earlier release of the program. What I did notice however, is that the default jamovi behaviour is assuming that homogeneity of variance (HOV) is violated, which goes against the basic assumption of the RM ANOVA. If HOV is present, then the same standard error (SE) should be used for the post hoc comparisons. If you look at the first post here (https://github.com/jamovi/jamovi/issues/660) and also on earlier discussions on the jamovi forum and compare it to the image below, you will notice that jamovi now uses different SE for the marginal means.

jamovi EMM RM ANOVA.jpg (24.28 KiB) Viewed 26423 times

What I put forward is that presenting the results based on different SE should not be the common output. In order to make an informed choice, I suggest that the choice need to be made by the user, according to the behaviour displayed by the data.

Regards,

[reason180]

I don't know that the issue ever concerned post-hoc tests. Rather, it has to do with the error variances for the the estimated marginal means.

[Mik]

I don't know that the issue ever concerned post-hoc tests. Rather, it has to do with the error variances for the the estimated marginal means.

Hi, yes it was partially related to the issue of post hoc tests,

"... lsmeans to calculate the post-hoc tests which uses the pooled variance from all groups that are being compared. Therefore, all standard errors have the same value. Using the pooled variance has some benefits such as a higher power, and it's a bit more robust when there are only a few counts in certain cells."

(viewtopic.php?t=114)

see also: https://forum.cogsci.nl/discussion/5794 ... thoc-tests where it is stated

"That's how the emmeans works: it operates based on the model you specify. In the ANOVA model, there is the assumption of equal variances, so that influences how emmeans handles the pairwise comparisons. We offer a workaround, where (for balanced designs only) in the RM ANOVA you can choose to use the non-pooled error term for RM factors."

[Ravi]

Hi!

I did a little digging and found the commit where this change was implemented (4 years ago!): https://github.com/jamovi/jmv/commit/a9 ... c60e5b2fd2. So instead of using a univariate model to calculate the emmeans, a multivariate model is used. This change was implemented back then because of recommendations made by the author of the package that we use to perform the RM ANOVA (see also this topic: viewtopic.php?t=1924).

So this change has been around for quite a while, but of course we can discuss whether an option to use the univariate model instead would be useful.

Cheers,
Ravi

[Mik]

Hi there,

Thanks for taking the time to dig about when the change was made in the procedure. While there is nothing wrong in using the multivariate model when the user is aware of the underlying reasons, I suggest that there would be an advantage when the choice is made by the user. If jamovi wants to put forth the more robust version of the RM ANOVA by default (like R, with the default t-test being the Welch t' test (assuming no equal variances) instead of the classic Student t test), I put forward that it would also be useful to have the univariate model available. When the cells in the design are balanced and when the sphericity assumption is respected, there is nothing wrong with the univariate model. I think that most data analysis courses will start with the basic model and then, more advanced data analysis courses will go on by relaxing assumptions. It would therefore be a benefit to have the option to select "consciously" the SE used by the RM ANOVA procedure for post-hoc tests and marginal means. Could it be possible to add that option in a subsequent release of jamovi ?

[Ravi]

Hi!

I think we can add this option to the RM ANOVA. Could you make an issue on github (this is where we keep track of feature requests): https://github.com/jamovi/jamovi/issues.

Cheers,
Ravi

[Mik]

Hello Ravi,

I just did it.

Thanks,

Michael