RM ANOVA - Post hoc Paired-sample t-test doesn't make sense

General help and assistance with jamovi. Bug reports can be made at our issues page: https://github.com/jamovi/jamovi/issues . (If you're unsure feel free to discuss it here)

by Renzo » Thu Feb 13, 2020 6:10 pm

Hi guys. First of all, congrats on such amazing software. If I were Bill Gates I would have donated you the money he spent on that boat he bought this week.

So, the problem I found: after finding a significant interaction in a 2 x 7 RM ANOVA, I ran Bonferroni-corrected paired-sample t-tests (post hoc window) and found results that don't make sense: t(149) = -3.672, p =.03 but the way I understand it, with Bonferroni correction, a t stat of 149 degrees of freedom and a value of 3.67 should yield a p value of .0003 and not .03. Is there something non-standard that this Bonferroni correction is doing?
Renzo
 
Posts: 2
Joined: Mon Mar 25, 2019 5:30 pm

by rheredia@tamiu.edu » Thu Feb 13, 2020 8:11 pm

Did you try the noncorrection feature? is the P = .03 from the noncorrected?
rheredia@tamiu.edu
 
Posts: 16
Joined: Fri Feb 09, 2018 3:41 pm

by Renzo » Thu Feb 13, 2020 8:15 pm

Yes, but I want the Bonferroni correction - it's just that the Bonferroni correction in Jamovi gives me a number that doesn't correspond to what the calculation of the Bonferroni correction (with other software and by hand) gives.

I thought it might have to do with the fact that Jamovi based its analyses on estimated marginal means, but I don't know...
Renzo
 
Posts: 2
Joined: Mon Mar 25, 2019 5:30 pm

by jonathon » Thu Feb 13, 2020 11:01 pm

hi,

"I ran Bonferroni-corrected paired-sample t-tests (post hoc window)"

so the post-hoc tests are based on the estimated marginal means, and aren't paired-samples t-tests per se. the post-hoc tests in jamovi are based on the model you are fitting, in this case, a repeated measures ANOVA. an independent set of paired-samples t-tests is a different model.

does that go some way to explaining the discrepancy?

the code for post-hocs is here:

https://github.com/jamovi/jmv/blob/mast ... #L586-L591

cheers

jonathon
User avatar
jonathon
 
Posts: 1405
Joined: Fri Jan 27, 2017 10:04 am

by RCL » Fri Feb 14, 2020 3:16 pm

Thank you, that's very clarifying. However, I don't understand what these post hoc comparisons are, conceptually. If they are based on EEMs and are Bonferroni corrected, what are they?
RCL
 
Posts: 13
Joined: Sat Apr 21, 2018 11:56 pm
Location: Edinburgh, United Kingdom

by jonathon » Wed Feb 19, 2020 7:49 am

here's an intro to emmeans:

https://cran.r-project.org/web/packages ... asics.html

the way i normally demonstrate emmeans, is to use the 'Tooth Growth' data set, run a two-way ANOVA (len ~ dose * supp), and plot the interaction. *then* remove the the interaction term from the model ... you'll see that this forces the lines in the interaction plot to be parallel. so this is what emmeans are ... the means estimated for each group, by the model you are fitting.

the post-hoc tests are, in turn, comparisons between these estimated means. if you think about it, it has to be this way - otherwise your post-hoc tests are unrelated to your ANOVA.

here's the wikipedia article on bonferroni correction:

https://en.wikipedia.org/wiki/Bonferroni_correction

let me know if you have any more questions.

cheers

jonathon
User avatar
jonathon
 
Posts: 1405
Joined: Fri Jan 27, 2017 10:04 am

by RCL » Thu Feb 20, 2020 2:51 am

Thanks so much, Jonathon. That was very helpful. I wonder if using emmeans is justified when you have an experimental design like in neuroscience experiments, where the data is very well balanced...
RCL
 
Posts: 13
Joined: Sat Apr 21, 2018 11:56 pm
Location: Edinburgh, United Kingdom

by jonathon » Thu Feb 20, 2020 4:52 am

yes, there's no problem using them with balanced data.

being able to use emmeans with unbalanced data is certainly advantage, but the bigger point here is that emmeans are based on the model you are fitting. they are based on the ANOVA.

if you want to fit a lot of separate t-tests, then it's appropriate to perform cohen's d on the descriptives, but if you're fitting a more complex model than that, then it makes sense that your cohen's d should be performed on the model.

jonathon
User avatar
jonathon
 
Posts: 1405
Joined: Fri Jan 27, 2017 10:04 am


Return to Help