Possible error in posthoc tests

Discuss the jamovi platform, possible improvements, etc.

by candice » Mon Apr 01, 2019 4:06 pm

Hi hi!

I ran a post hoc test on an interaction and I'm getting strange results.

As I understand it (which may be wrong), the paired comparisons should be giving me the same t values, degrees of freedom, and standard errors as running the paired samples t-tests, right? It's giving me different results for the post hoc test and the paired samples t-tests, even for the uncorrected post hoc.

Is there something I'm missing here?
I ran it using other programs and I got different results.

I attached the file with the data and analyses as well as the output, in case I'm the only one getting something strange.

Thank you!!!
(2.33 KiB) Downloaded 43 times
dice jonan.omv
data + analysis
(11.83 KiB) Downloaded 32 times
Posts: 3
Joined: Mon Apr 01, 2019 3:30 pm

by jonathon » Mon Apr 01, 2019 11:24 pm


the post-hoc tests in jamovi as based on estimated marginal means, and aren't based on paired samples t-tests. the advantage of this approach is that you're using the *same* model for the post hoc tests, as you are for the original model (it also has some benefits wrt missing values too).

i think it's worth thinking about which model is right for this data. is an ANOVA appropriate, or separate t-tests? and then use one or the other.

User avatar
Posts: 841
Joined: Fri Jan 27, 2017 10:04 am

by candice » Tue Apr 02, 2019 9:32 am

Right, but then why is it giving me strange degrees of freedom? I also ran the same test with different data and didn't have the same problem.

I'm sorry, I'm not familiar with that way of doing a posthoc test, do you have any resources I could use to read up on this and understand it better?

Thank you so much!!!
Posts: 3
Joined: Mon Apr 01, 2019 3:30 pm

by jonathon » Wed Apr 03, 2019 1:49 am


i've always been on the look out for a good article on this, but i haven't found one yet. but basically, the post-hoc tests are carried out on the estimated marginal means, rather than the marginal means (descriptives).

this article outlines the advantages of emmeans:

https://cran.r-project.org/web/packages ... asics.html

User avatar
Posts: 841
Joined: Fri Jan 27, 2017 10:04 am

by reason180 » Wed Apr 03, 2019 9:39 am

Individual t tests are not going to be using variances pooled across more than two groups, like an ANOVA model does. So that's one difference. But regarding the the unexpected degrees of freedom, I don't think the issue is that estimated marginal means are being used, per se. Instead, I think the issue is the same as what was addressed in another fairly recent thread (that actually spanned across multiple threads in multiple forums--but here's the recent jamovi thread: https://forum.jamovi.org/viewtopic.php?f=2&t=689 ).

As a understand it, the conclusions of that discussion are summarized as follows:

(1) For ANOVA post-hoc tests, jamovi uses Welch's t tests, not Student's t tests.(See https://en.wikipedia.org/wiki/Welch%27s_t-test ).

(2) In the case of repeated-measures ANOVA, the full Welch method is used, including the Welch-Satterthwaite method for computing degrees of freedom for the post-hoc test. In that method, the degrees of freedom are influenced not just by the number of observations, but also by the observed variances. Again, see https://en.wikipedia.org/wiki/Welch%27s_t-test .

(3) For the non-repeated-measures ANOVA, the post-hocs also use Welch's rather than Student's t, but it is a variation of the Welch method. In this variation, all the elements of the Welch method are used except that the Welch-Satterthwaite degrees of freedom are not used. Instead, degrees of freedom are calculated the way one would calculate them for Student's t.

(4) The differences in the way degrees of freedom are calculated in the repeated-measures ANOVA post-hocs versus the non-repeated-measures ANOVA post-hocs is due to the fact that jamovi's ANOVA routines make use of the EMMEANS package in R, and that the different R programmers who created the packages on which EMMEANS relies made different choices about how to calculate the degrees of freedom for the Welch's t tests.
Last edited by reason180 on Sat Apr 13, 2019 3:32 am, edited 3 times in total.
Posts: 26
Joined: Mon Jul 24, 2017 4:56 pm

by candice » Wed Apr 03, 2019 9:45 am

Thank you so much for the very thorough reply!!
I think I get it now. Thank you so so so much!

Posts: 3
Joined: Mon Apr 01, 2019 3:30 pm

Return to General