Cohen's d in post-hoc test

Discuss the jamovi platform, possible improvements, etc.
Post Reply
Vicente_inefo
Posts: 16
Joined: Wed May 29, 2019 11:39 pm

Cohen's d in post-hoc test

Post by Vicente_inefo »

Hi,

I have performed a mixed 2x2 ANOVA and I noticed that the post-hoc test does not report the effect size. Is there any possibility to implement it?

On the other hand, if JAMOVI computes Cohen's d by the formula d = t/sqrt(n), it would be correct to compute Cohen's d by using the t value reported by JAMOVI and the n value of the respective group?

Please, see the attached file for more details

I accept any other suggestion, thanks in advance
Attachments
Post-hoc test table
Post-hoc test table
Captura de pantalla 2019-10-25 a las 22.12.43.png (115.35 KiB) Viewed 13415 times
User avatar
jonathon
Posts: 2609
Joined: Fri Jan 27, 2017 10:04 am

Re: Cohen's d in post-hoc test

Post by jonathon »

hi,

the issue is that the post-hoc tests in jamovi are based on the estimated marginal means, rather than simply being pairwise comparisons of the raw data. so calculating cohen's d for these is a bit more complicated ... i *think* it's possible to calculate cohen's d for emmeans, but even then i'm not completely sure.

i've asked a few people about it, and i'll see what they come back with.

cheers
Vicente_inefo
Posts: 16
Joined: Wed May 29, 2019 11:39 pm

Re: Cohen's d in post-hoc test

Post by Vicente_inefo »

Thank you very much, I will remain attentive to your response
jose.lopez
Posts: 1
Joined: Mon Feb 10, 2020 11:52 am

Re: Cohen's d in post-hoc test

Post by jose.lopez »

Hi,

I was also confused to find that the Cohen's d calculated with the post-hoc comparisons does not match the classical (y1-y2)/S formula. It was useful to learn from Jonathon that the values are based on estimated marginal means. However, I'd still be interested in Cohen's d based on the observed means for teaching purposes, so I'm wondering if anyone can help me find an option to obtain these with jamovi?

Cheers,
José
User avatar
jonathon
Posts: 2609
Joined: Fri Jan 27, 2017 10:04 am

Re: Cohen's d in post-hoc test

Post by jonathon »

hey,

yeah, we'd like to provide a 'pairwise' comparisons module, that works on descriptives, rather than emms. don't get me wrong, i think working off descriptives is wrong most of the time :P but it seems to be something a lot of people want.

cheers

jonathon
User avatar
reason180
Posts: 268
Joined: Mon Jul 24, 2017 4:56 pm

Re: Cohen's d in post-hoc test

Post by reason180 »

It may be that people are skeptical of emmeans because they're not sure exactly how the estimated marginals relate to the regular old marginals.

I think in the case of an ANOVA, the estimated marginal means are identical to the marginal means UNLESS you're averaging over the levels of a know factor such that the averaged-over levels have unequal sample sizes. In the latter case, the estimated marginal mean are the means of the averaged-over means, rather than the means of raw data. Thus for example, if there are scores for left- and right-handed men and women, and if there are 10 left-handed and 90 right-handed participants, and if the ANOVA model includes handedness and gender as factors, and if one is interested in looking just at the main effect of gender then:

The estimated marginal means for the main effect of gender are not the mean scores for the women versus the mean scores for the men. Rather they are the mean of two means--the mean score for left-handed women and the mean score for right-handed women--versus the mean of two other means--the mean score for left-handed men and the mean score for right-handed men. Thus the estimated marginal means are the means that one would find had the study included an equal number of left- and right-handed participants.

But I don't know how to calculate the variances that correspond to those estimated marginal means (when the sizes of the averaged samples are unequal). This is because it's more complicated: It involves not just pooling variances (consistent with the equal-variance assumption) but also modeling what the variances would be had the sample sizes for the averaged categories been equal.

(Of course, all of this is even more complicated when the dealing with ANCOVA.)
User avatar
mcfanda@gmail.com
Posts: 452
Joined: Thu Mar 23, 2017 9:24 pm

Re: Cohen's d in post-hoc test

Post by mcfanda@gmail.com »

The easiest way is to use the t-test. d=t*sqrt(1/n1+1/n2)
Vicente_inefo
Posts: 16
Joined: Wed May 29, 2019 11:39 pm

Re: Cohen's d in post-hoc test

Post by Vicente_inefo »

mcfanda@gmail.com wrote:The easiest way is to use the t-test. d=t*sqrt(1/n1+1/n2)
I was talking about a within subject effect size (i.e. pre-test vs post-test). I think that JAMOVI computes Cohen's d of the paired samples t-test using the formula d = t/sqrt(n).

In this case, it would be correct to compute Cohen's d by using the t value reported by JAMOVI and the n value of the respective pairs of groups? O maybe, simply use the Cohen's d computed in a paired t-test executed in a different analysis?

Thanks in advance
User avatar
jonathon
Posts: 2609
Joined: Fri Jan 27, 2017 10:04 am

Re: Cohen's d in post-hoc test

Post by jonathon »

yup. if you want to calculate a cohen's d for the mean of the difference between two measurements (i.e. a paired samples design), then that's a paired-samples t-test model, and you should use the cohen's d estimated in the paired samples t-test.

the cohen's d we report, is always appropriate for the model you are fitting.

if you want a different cohen's d, you want a different model.

jonathon
philifide
Posts: 1
Joined: Tue May 17, 2022 1:20 pm

Re: Cohen's d in post-hoc test

Post by philifide »

...calculating cohen's d for these is a bit more complicated ... i *think* it's possible to calculate cohen's d for emmeans, but even then i'm not completely sure.

i've asked a few people about it, and i'll see what they come back with.
Apologies for the thread necro, but did you get any responses Jonathon? Just curious if anyone suggested a means of calculating an effect size with emmeans (independent or dependent groups).

I think the emmeans package in R has a method of working it out, but I'm not sure what their formula is: https://cran.r-project.org/web/packages ... isons.html
Post Reply