Possible error in posthoc tests

Discuss the jamovi platform, possible improvements, etc.
candice
Posts: 3
Joined: Mon Apr 01, 2019 3:30 pm

Possible error in posthoc tests

Post by candice »

Hi hi!

I ran a post hoc test on an interaction and I'm getting strange results.

As I understand it (which may be wrong), the paired comparisons should be giving me the same t values, degrees of freedom, and standard errors as running the paired samples t-tests, right? It's giving me different results for the post hoc test and the paired samples t-tests, even for the uncorrected post hoc.

Is there something I'm missing here?
I ran it using other programs and I got different results.

I attached the file with the data and analyses as well as the output, in case I'm the only one getting something strange.

Thank you!!!
Attachments
output.csv
output
(2.33 KiB) Downloaded 503 times
dice jonan.omv
data + analysis
(11.83 KiB) Downloaded 510 times
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Possible error in posthoc tests

Post by jonathon »

hi,

the post-hoc tests in jamovi as based on estimated marginal means, and aren't based on paired samples t-tests. the advantage of this approach is that you're using the *same* model for the post hoc tests, as you are for the original model (it also has some benefits wrt missing values too).

i think it's worth thinking about which model is right for this data. is an ANOVA appropriate, or separate t-tests? and then use one or the other.

jonathon
candice
Posts: 3
Joined: Mon Apr 01, 2019 3:30 pm

Re: Possible error in posthoc tests

Post by candice »

Right, but then why is it giving me strange degrees of freedom? I also ran the same test with different data and didn't have the same problem.

I'm sorry, I'm not familiar with that way of doing a posthoc test, do you have any resources I could use to read up on this and understand it better?

Thank you so much!!!
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Possible error in posthoc tests

Post by jonathon »

hey,

i've always been on the look out for a good article on this, but i haven't found one yet. but basically, the post-hoc tests are carried out on the estimated marginal means, rather than the marginal means (descriptives).

this article outlines the advantages of emmeans:

https://cran.r-project.org/web/packages ... asics.html

jonathon
User avatar
reason180
Posts: 274
Joined: Mon Jul 24, 2017 4:56 pm

Re: Possible error in posthoc tests

Post by reason180 »

Individual t tests are not going to be using variances pooled across more than two groups, like an ANOVA model does. So that's one difference. But regarding the the unexpected degrees of freedom, I don't think the issue is that estimated marginal means are being used, per se. Instead, I think the issue is the same as what was addressed in another fairly recent thread (that actually spanned across multiple threads in multiple forums--but here's the recent jamovi thread: viewtopic.php?f=2&t=689 ).

As a understand it, the conclusions of that discussion are summarized as follows:

(1) For ANOVA post-hoc tests, jamovi uses Welch's t tests, not Student's t tests.(See https://en.wikipedia.org/wiki/Welch%27s_t-test ).

(2) In the case of repeated-measures ANOVA, the full Welch method is used, including the Welch-Satterthwaite method for computing degrees of freedom for the post-hoc test. In that method, the degrees of freedom are influenced not just by the number of observations, but also by the observed variances. Again, see https://en.wikipedia.org/wiki/Welch%27s_t-test .

(3) For the non-repeated-measures ANOVA, the post-hocs also use Welch's rather than Student's t, but it is a variation of the Welch method. In this variation, all the elements of the Welch method are used except that the Welch-Satterthwaite degrees of freedom are not used. Instead, degrees of freedom are calculated the way one would calculate them for Student's t.

(4) The differences in the way degrees of freedom are calculated in the repeated-measures ANOVA post-hocs versus the non-repeated-measures ANOVA post-hocs is due to the fact that jamovi's ANOVA routines make use of the EMMEANS package in R, and that the different R programmers who created the packages on which EMMEANS relies made different choices about how to calculate the degrees of freedom for the Welch's t tests.
Last edited by reason180 on Sat Apr 13, 2019 3:32 am, edited 3 times in total.
candice
Posts: 3
Joined: Mon Apr 01, 2019 3:30 pm

Re: Possible error in posthoc tests

Post by candice »

Thank you so much for the very thorough reply!!
I think I get it now. Thank you so so so much!

Cheers!
Sjoerd.Bruijn
Posts: 2
Joined: Tue Apr 07, 2020 11:17 am

Re: Possible error in posthoc tests

Post by Sjoerd.Bruijn »

Hi,

I too was wondering about the posthoc. I was confused by the SE being the same for all posthoc comparisons after a RM ANOVA, but I understand now that this is due to the way the posthoc testing is implemented, which is based on EM, is that correct? Nonentheless, for plotting the same results, the error bars are also the same size for each condition, which seems rather odd, especially if the underlying data clearly shows difference is variance between the conditions. Is this considered to be a bug, or a feature (or both? ). To me, it would seem logical that the error bars are different for different conditions?

Btw, by no means a critique, as I love your software, great package!
S
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Possible error in posthoc tests

Post by jonathon »

i think you've pretty squarely hit the nail on the head here:
Nonetheless, for plotting the same results, the error bars are also the same size for each condition, which seems rather odd, especially if the underlying data clearly shows difference is variance between the conditions
the underlying data may have differences in variance, but *the model* doesn't. RM ANOVA assumes equal variance. this is what's key here, the post-hoc tests are based on the model that you're fitting. if your model doesn't allow for different variances between groups, then neither will the post-hoc tests.

and yup, this is all a consequence of them being based on the estimated marginal means.

cheers

jonathon
Sjoerd.Bruijn
Posts: 2
Joined: Tue Apr 07, 2020 11:17 am

Re: Possible error in posthoc tests

Post by Sjoerd.Bruijn »

ok, so it's basically the model you're plotting, not really the data. Got it. Thanks!
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Possible error in posthoc tests

Post by jonathon »

it's both. it's the model (and all the assumptions that come with that model), fitted to the data :)

jonathon
Post Reply