jamovi

Hi hi!

I ran a post hoc test on an interaction and I'm getting strange results.

As I understand it (which may be wrong), the paired comparisons should be giving me the same t values, degrees of freedom, and standard errors as running the paired samples t-tests, right? It's giving me different results for the post hoc test and the paired samples t-tests, even for the uncorrected post hoc.

Is there something I'm missing here?
I ran it using other programs and I got different results.

I attached the file with the data and analyses as well as the output, in case I'm the only one getting something strange.

Thank you!!!

hi,

the post-hoc tests in jamovi as based on estimated marginal means, and aren't based on paired samples t-tests. the advantage of this approach is that you're using the *same* model for the post hoc tests, as you are for the original model (it also has some benefits wrt missing values too).

i think it's worth thinking about which model is right for this data. is an ANOVA appropriate, or separate t-tests? and then use one or the other.

jonathon

Right, but then why is it giving me strange degrees of freedom? I also ran the same test with different data and didn't have the same problem.

I'm sorry, I'm not familiar with that way of doing a posthoc test, do you have any resources I could use to read up on this and understand it better?

Thank you so much!!!

hey,

i've always been on the look out for a good article on this, but i haven't found one yet. but basically, the post-hoc tests are carried out on the estimated marginal means, rather than the marginal means (descriptives).

this article outlines the advantages of emmeans:

https://cran.r-project.org/web/packages ... asics.html

jonathon

Individual t tests are not going to be using variances pooled across more than two groups, like an ANOVA model does. So that's one difference. But regarding the the unexpected degrees of freedom, I don't think the issue is that estimated marginal means are being used, per se. Instead, I think the issue is the same as what was addressed in another fairly recent thread (that actually spanned across multiple threads in multiple forums--but here's the recent jamovi thread: viewtopic.php?f=2&t=689 ).

As a understand it, the conclusions of that discussion are summarized as follows:

(1) For ANOVA post-hoc tests, jamovi uses Welch's t tests, not Student's t tests.(See https://en.wikipedia.org/wiki/Welch%27s_t-test ).

(2) In the case of repeated-measures ANOVA, the full Welch method is used, including the Welch-Satterthwaite method for computing degrees of freedom for the post-hoc test. In that method, the degrees of freedom are influenced not just by the number of observations, but also by the observed variances. Again, see https://en.wikipedia.org/wiki/Welch%27s_t-test .

(3) For the non-repeated-measures ANOVA, the post-hocs also use Welch's rather than Student's t, but it is a variation of the Welch method. In this variation, all the elements of the Welch method are used except that the Welch-Satterthwaite degrees of freedom are not used. Instead, degrees of freedom are calculated the way one would calculate them for Student's t.

(4) The differences in the way degrees of freedom are calculated in the repeated-measures ANOVA post-hocs versus the non-repeated-measures ANOVA post-hocs is due to the fact that jamovi's ANOVA routines make use of the EMMEANS package in R, and that the different R programmers who created the packages on which EMMEANS relies made different choices about how to calculate the degrees of freedom for the Welch's t tests.

Thank you so much for the very thorough reply!!
I think I get it now. Thank you so so so much!

Cheers!

Hi,

I too was wondering about the posthoc. I was confused by the SE being the same for all posthoc comparisons after a RM ANOVA, but I understand now that this is due to the way the posthoc testing is implemented, which is based on EM, is that correct? Nonentheless, for plotting the same results, the error bars are also the same size for each condition, which seems rather odd, especially if the underlying data clearly shows difference is variance between the conditions. Is this considered to be a bug, or a feature (or both? ). To me, it would seem logical that the error bars are different for different conditions?

Btw, by no means a critique, as I love your software, great package!
S

i think you've pretty squarely hit the nail on the head here:

Nonetheless, for plotting the same results, the error bars are also the same size for each condition, which seems rather odd, especially if the underlying data clearly shows difference is variance between the conditions

the underlying data may have differences in variance, but *the model* doesn't. RM ANOVA assumes equal variance. this is what's key here, the post-hoc tests are based on the model that you're fitting. if your model doesn't allow for different variances between groups, then neither will the post-hoc tests.

and yup, this is all a consequence of them being based on the estimated marginal means.

cheers

jonathon

ok, so it's basically the model you're plotting, not really the data. Got it. Thanks!

it's both. it's the model (and all the assumptions that come with that model), fitted to the data

jonathon