Major inconsistency in RM ANOVA Pairwise comparisons!

General help and assistance with jamovi. Bug reports can be made at our issues page: https://github.com/jamovi/jamovi/issues . (If you're unsure feel free to discuss it here)
Post Reply
RCL
Posts: 16
Joined: Sat Apr 21, 2018 11:56 pm
Location: Edinburgh, United Kingdom

Major inconsistency in RM ANOVA Pairwise comparisons!

Post by RCL »

Hi,
I have a 7-experiment dataset that I analysed using Jamovi. Most of the experiments' data were analysed by using RM ANOVAs and Bonferroni-corrected paired tests whenever there was a significant interaction. I wrote the manuscript using those results. Now, I needed to run a new analysis and I found that post hoc tests are giving *VERY* different results than months ago.

Two things I've noticed: the Post hoc tests are giving very different degrees of freedom now. For 32 subjects, now they give 31 df whereas they were giving around 140 df months ago. I guess you made a major change in how you are calculating post hoc tests.

1. What did you change and why?
2. What version of Jamovi should I download if I want post hoc tests to be calculated as before?
3. Is it reliable to run post hoc tests in Jamovi? Things have changed several times the past year and to be honest, I don't know if Jamovi is a reliable stats package anymore. This time, the changes in the results were extreme.

Thanks.
RCL
Posts: 16
Joined: Sat Apr 21, 2018 11:56 pm
Location: Edinburgh, United Kingdom

Re: Major inconsistency in RM ANOVA Pairwise comparisons!

Post by RCL »

I can't find any older versions of Jamovi! :(
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Major inconsistency in RM ANOVA Pairwise comparisons!

Post by jonathon »

your best bet for looking for changes to analyses is to look at the history of changes to the appropriate .b.R file in the jmv repo.

https://github.com/jamovi/jmv/commits/master/R/anovarm.b.R

you'll see we moved from:

emmeans::emmeans(result, formula)

to:

emmeans::emmeans(result, formula, model="multivariate")

you'll find some details of the change here:

https://forum.jamovi.org/viewtopic.php?f=2&t=1646&p=5920#p5915

the afex package is also moving to model="multivariate" too

https://cran.r-project.org/web/packages/afex/vignettes/afex_anova_example.html#data-and-r-preperation

i assume this is the source of the discrepancy.

cheers

jonathon
Post Reply