RM ANOVA - Post hoc Paired-sample t-test doesn't make sense

General help and assistance with jamovi. Bug reports can be made at our issues page: https://github.com/jamovi/jamovi/issues . (If you're unsure feel free to discuss it here)
Post Reply
Renzo
Posts: 2
Joined: Mon Mar 25, 2019 5:30 pm

RM ANOVA - Post hoc Paired-sample t-test doesn't make sense

Post by Renzo »

Hi guys. First of all, congrats on such amazing software. If I were Bill Gates I would have donated you the money he spent on that boat he bought this week.

So, the problem I found: after finding a significant interaction in a 2 x 7 RM ANOVA, I ran Bonferroni-corrected paired-sample t-tests (post hoc window) and found results that don't make sense: t(149) = -3.672, p =.03 but the way I understand it, with Bonferroni correction, a t stat of 149 degrees of freedom and a value of 3.67 should yield a p value of .0003 and not .03. Is there something non-standard that this Bonferroni correction is doing?
rheredia@tamiu.edu
Posts: 41
Joined: Fri Feb 09, 2018 3:41 pm

Re: RM ANOVA - Post hoc Paired-sample t-test doesn't make se

Post by rheredia@tamiu.edu »

Did you try the noncorrection feature? is the P = .03 from the noncorrected?
Renzo
Posts: 2
Joined: Mon Mar 25, 2019 5:30 pm

Re: RM ANOVA - Post hoc Paired-sample t-test doesn't make se

Post by Renzo »

Yes, but I want the Bonferroni correction - it's just that the Bonferroni correction in Jamovi gives me a number that doesn't correspond to what the calculation of the Bonferroni correction (with other software and by hand) gives.

I thought it might have to do with the fact that Jamovi based its analyses on estimated marginal means, but I don't know...
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: RM ANOVA - Post hoc Paired-sample t-test doesn't make se

Post by jonathon »

hi,

"I ran Bonferroni-corrected paired-sample t-tests (post hoc window)"

so the post-hoc tests are based on the estimated marginal means, and aren't paired-samples t-tests per se. the post-hoc tests in jamovi are based on the model you are fitting, in this case, a repeated measures ANOVA. an independent set of paired-samples t-tests is a different model.

does that go some way to explaining the discrepancy?

the code for post-hocs is here:

https://github.com/jamovi/jmv/blob/mast ... #L586-L591

cheers

jonathon
RCL
Posts: 16
Joined: Sat Apr 21, 2018 11:56 pm
Location: Edinburgh, United Kingdom

Re: RM ANOVA - Post hoc Paired-sample t-test doesn't make se

Post by RCL »

Thank you, that's very clarifying. However, I don't understand what these post hoc comparisons are, conceptually. If they are based on EEMs and are Bonferroni corrected, what are they?
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: RM ANOVA - Post hoc Paired-sample t-test doesn't make se

Post by jonathon »

here's an intro to emmeans:

https://cran.r-project.org/web/packages ... asics.html

the way i normally demonstrate emmeans, is to use the 'Tooth Growth' data set, run a two-way ANOVA (len ~ dose * supp), and plot the interaction. *then* remove the the interaction term from the model ... you'll see that this forces the lines in the interaction plot to be parallel. so this is what emmeans are ... the means estimated for each group, by the model you are fitting.

the post-hoc tests are, in turn, comparisons between these estimated means. if you think about it, it has to be this way - otherwise your post-hoc tests are unrelated to your ANOVA.

here's the wikipedia article on bonferroni correction:

https://en.wikipedia.org/wiki/Bonferroni_correction

let me know if you have any more questions.

cheers

jonathon
RCL
Posts: 16
Joined: Sat Apr 21, 2018 11:56 pm
Location: Edinburgh, United Kingdom

Re: RM ANOVA - Post hoc Paired-sample t-test doesn't make se

Post by RCL »

Thanks so much, Jonathon. That was very helpful. I wonder if using emmeans is justified when you have an experimental design like in neuroscience experiments, where the data is very well balanced...
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: RM ANOVA - Post hoc Paired-sample t-test doesn't make se

Post by jonathon »

yes, there's no problem using them with balanced data.

being able to use emmeans with unbalanced data is certainly advantage, but the bigger point here is that emmeans are based on the model you are fitting. they are based on the ANOVA.

if you want to fit a lot of separate t-tests, then it's appropriate to perform cohen's d on the descriptives, but if you're fitting a more complex model than that, then it makes sense that your cohen's d should be performed on the model.

jonathon
Post Reply