Different Cohen's d depending on App or R

General help and assistance with jamovi. Bug reports can be made at our issues page: https://github.com/jamovi/jamovi/issues . (If you're unsure feel free to discuss it here)
Post Reply
hjin
Posts: 5
Joined: Wed Jun 23, 2021 2:06 am

Different Cohen's d depending on App or R

Post by hjin »

I'm not sure if this has been posted before but I noticed that getting the effect size for pairwise comparisons in ANOVA directly through the jamovi program yields different results from running the exact same analysis through R.

For instance, one of the pairwise comparisons has VAR1 (n = 75) with M ± SD = 6.8 ± 3.5 and VAR2 (n = 74) with M ± SD = 4.5 ± 3.0, which comes out to be approximately a cohen's d of ~0.7. This seems to be calculated correctly in the jamovi application but when I run the same analysis through R, the cohen's d comes out to be around around d = 0.25. All other metrics (e.g., p-value, test statistic, etc.) are the same, however.

From doing a bit of calculation, it looks like the ANOVA cohen's d calculation when done through R is dividing the Cohen's d calculation by a constant of 2.81898 (I have a 2x2 factorial two-way ANOVA). Is this the program in R erroneously applying a Bonferroni-style adjustment to the Cohen's d?

*I'm using jamovi v.1.8.1.0 and I've updated the jmv/jmvcore packages on R but am still getting the same issue. I ran the same analysis through the jamovi program's R editor module and it's coming out correctly, so it seems isolated to just R.
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Different Cohen's d depending on App or R

Post by jonathon »

for post-hoc tests, in jamovi, the cohen's d are based on the estimated marginal means, which might be different to what you're doing in R?

jonathon
hjin
Posts: 5
Joined: Wed Jun 23, 2021 2:06 am

Re: Different Cohen's d depending on App or R

Post by hjin »

Hi Jonathon,

Thanks for the response! Sorry, to clarify, I'm running the exact same jmv ANOVA on R using the jmv syntax (where I copy the syntax that the jamovi program gives me and paste it in R which makes it a bit easier for just record purposes).

So when I run the analysis in the jamovi program, I get the following:
jamovi1.JPG
jamovi1.JPG (13.74 KiB) Viewed 7201 times
But when I run the same analysis using the jmv syntax in R, I get the following:
jamovi2.JPG
jamovi2.JPG (19.21 KiB) Viewed 7201 times
Note that the p-values are the same but the effect sizes are showing up different for some reason.


I'm using the following syntax:

jmv::ANOVA(
formula = dv ~ iv1 + iv2 + iv1:iv2,
data = data,
effectSize = "partEta",
postHoc = ~ iv1:iv2,
postHocES = "d",
emMeans = ~ iv1:iv2
)
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Different Cohen's d depending on App or R

Post by jonathon »

oh yup, update your version of jmv and it will fix it. there's all these additional requirements that CRAN have imposed on R packages, and it's made it difficult for us to update the version on CRAN.

you can install the latest version with:

remotes::install_github('jamovi/jmv')

cheers

jonathon
hjin
Posts: 5
Joined: Wed Jun 23, 2021 2:06 am

Re: Different Cohen's d depending on App or R

Post by hjin »

Ah I see now that the one from CRAN is v.1.2 instead of the newest one!
I updated the version to the newest version but now I'm getting the following error:

Error: 'OptionOutput' is not an exported object from 'namespace:jmvcore'

I'm not sure what the problem can be,
I tried multiple things, like changing all my variables to numeric, or changing the class of my grouping variables to factors, and got rid of every additional command in the syntax but jmv::ANOVA won't even run with just the formula and data commands.

Other analyses (e.g., correlation, descriptives, etc.) seem to be running fine though so I'm not sure if something else is wrong on my end.
User avatar
jonathon
Posts: 2613
Joined: Fri Jan 27, 2017 10:04 am

Re: Different Cohen's d depending on App or R

Post by jonathon »

hey, you need to update your jmvcore too (sorry, i forgot to mention):

remotes::install_github('jamovi/jmvcore')
hjin
Posts: 5
Joined: Wed Jun 23, 2021 2:06 am

Re: Different Cohen's d depending on App or R

Post by hjin »

Everything's coming out great now, thank you Jonathon!
User avatar
MAgojam
Posts: 421
Joined: Thu Jun 08, 2017 2:33 pm
Location: Parma (Italy)

Re: Different Cohen's d depending on App or R

Post by MAgojam »

Hi, @hjin.
Your report reminded me of a similar thing, which happened some time ago to a colleague who asked me for help.
I had checked using Rj with "jamovi R" and "System R" on my PC without highlighting any differences, while this was happening on the colleague's PC.
The differences highlighted in the two R environments were due to two becoming versions of jmv.
The colleague updated jmv from CRAN with a later version than the one present in the last jamovi used.
I suggest you check the version you have in system R and if different from jamovi R, update from github.
You should have something like in the attachment.
screenshot.png
screenshot.png (34.17 KiB) Viewed 7168 times
If the problem doesn't go away, it comes back.

Cheers,
Maurizio

---------------------
Sorry, Jonathon.
The page did not update and I only realized now that you had already suggested the solution.

Cheers,
Maurizio
Post Reply