Comparing two effect sizes

Discuss statistics related things
Vit
Posts: 65
Joined: Fri Apr 19, 2019 11:51 am

Comparing two effect sizes

Post by Vit »

Hi all,
I want to consult a problem.

I am consulting a student who runs a study interested in the effect of supplementation on physical performance (peak force) in two exercises.

In a within-subject design, a group of participants will perform each exercise under a) no supplementation, b) placebo, c) low dose, d) high dose (a,b,c,d states double-blinded and randomized).

The peak forces between the two exercises differ greatly.

The plan is to use one-way RM ANOVA for each exercise and see whether supplementation affects performance and how large it is.

But it is also interesting and needs to be known whether the observed effect sizes differ between exercises, i.e. 'Does the effect of high dose in exercise 1 differ from the effect of high dose in exercise 2?'

What would you suggest as an appropriate approach using Jamovi?
Vit
Posts: 65
Joined: Fri Apr 19, 2019 11:51 am

Re: Comparing two effect sizes

Post by Vit »

Here is a dataset with simulated data (exaggerating the effects...) to play with.
Attachments
Dummy dataset.omv
(47.59 KiB) Downloaded 123 times
User avatar
MAgojam
Posts: 421
Joined: Thu Jun 08, 2017 2:33 pm
Location: Parma (Italy)

Re: Comparing two effect sizes

Post by MAgojam »

Hey Vit,
I would suggest a mixed Anova with the mixed model of the GAMLj module for analysis of a simple repeated measures design with an within-subject factor and a between-subject factor.
Attached is the supplement.omv file with the data of your example in the long format.
supplement.omv
(42.99 KiB) Downloaded 128 times
Cheers,
Maurizio
Vit
Posts: 65
Joined: Fri Apr 19, 2019 11:51 am

Re: Comparing two effect sizes

Post by Vit »

Hi,
Thanks for your reply. Yeah, I see. Mixed effect models are a bit away from the current students' curriculum, and it is difficult to run a power analysis.

If the exercise were comparable, this is the way I would go. But when participants lift on average 50kg in exercise 1 and 100 kg in exercise 2.

As far as I understand, the post hoc comparison you suggest does not seem to help. Or am I missing something? It only shows whether means differ (and they do without the effect of supplementation).

I want to compare effects. An imaginary example: If a high dose (compared to placebo) improves exercise 1 with Cohen's d = 0.5 [0.1, 0.9] and in exercise 2 with Cohen's d = 0.4 [0.0, 0.8], how can I formally compare the two Cohen's ds?

Only looking at whether CIs overlap does not seem to cut it.
User avatar
MAgojam
Posts: 421
Joined: Thu Jun 08, 2017 2:33 pm
Location: Parma (Italy)

Re: Comparing two effect sizes

Post by MAgojam »

Vit wrote:Only looking at whether CIs overlap does not seem to cut it.
Looking at the overlap of individual confidence intervals is not sound. I think you should calculate a confidence interval for the "difference of mean differences" and check if this includes 0. But this is done in principle by testing the appropriate linear hypothesis.

It is already in your readings this:
https://journals.sagepub.com/doi/abs/10 ... lCode=epma
Since you are interested in moving into jamovi, have you thought about looking at the module "esci - Effect Sizes and Confidece Intervals for R and Jamovi" v0.9.4 by Robert J Calin-Jageman?

Cheers,
Maurizio
Vit
Posts: 65
Joined: Fri Apr 19, 2019 11:51 am

Re: Comparing two effect sizes

Post by Vit »

Thank you for the suggestion of difference of mean differences and article. I'll read it.

I tried using the ESCI module (which I like). See attached file. To get CI for the 'difference of mean differences', I tried computing the difference between high dose and placebo for Excercise 1 and difference for high dose and placebo for exercise 2. I compared these two new columns using the Estimate paired mean difference in ESCI.

Is it correct? Sorry for being this dull.


It seems this should not be difficult to do, but I struggle to find much info.
Attachments
Dummy dataset.omv
(88.39 KiB) Downloaded 122 times
User avatar
MAgojam
Posts: 421
Joined: Thu Jun 08, 2017 2:33 pm
Location: Parma (Italy)

Re: Comparing two effect sizes

Post by MAgojam »

Hey Vit,
I would say yes, but I suggest you double check the "Exercise1 High-Placebo difference" variable in your sample data, because it seems that "LOW DOSE" has been subtracted from "HIGH DOSE" and not "PLACEBO".
You took a look here:
https://blog.jamovi.org/2020/06/09/esci.html
https://www.tandfonline.com/doi/epub/10 ... ccess=true

You might also be interested in looking in the jamovi library for Daniel Lakens' TOSTER module.
https://journals.sagepub.com/doi/pdf/10 ... 0617697177
http://daniellakens.blogspot.com/2016/1 ... ckage.html
https://journals.sagepub.com/doi/pdf/10 ... 5918770963
https://academic.oup.com/psychsocgeront ... ogin=false
TOSTER in development (updated) with respect to the module you can find it here:
https://github.com/Lakens/TOSTER

Cheers,
Maurizio
Vit
Posts: 65
Joined: Fri Apr 19, 2019 11:51 am

Re: Comparing two effect sizes

Post by Vit »

Oh shoot, I am glad I shared my data openly and transparently for others to reproduce and check my errors:-). Luckily, this error does not affect the theoretical question at hand.

Yes, I considered Equivalence testing. But still, I feel all these ESCI and TOSTER approaches focus on comparing means, not effect sizes. It is evident here the two exercises differ by default, no matter the intervention.

I guess this might be an issue for meta-analytical approaches.
e.g. https://stats.stackexchange.com/questio ... -cohens-ds

I'll check using the Major module or the by hand approach from the link above.
Vit
Posts: 65
Joined: Fri Apr 19, 2019 11:51 am

Re: Comparing two effect sizes

Post by Vit »

Using MAJOR module and Effect size and Sample variation analysis should do the job base on Q and associated p-value.

I calculated Variance for each effect as Vdav=((CIupper-CIlower)/2*1.96)^2
https://www.researchgate.net/post/Calcu ... es-designs

But I got a known MAJOR module error "Y is empty or has only NAs". This error happens when only 2 effects are entered. It works with >3 effects. This error should have been resolved in new versions (I am on Jamovi 2.3.0 and current MAJOR 1.2.1).

It is getting discouraging how such a simple question si challenging to answer, and in the last step, there is an error message.
User avatar
MAgojam
Posts: 421
Joined: Thu Jun 08, 2017 2:33 pm
Location: Parma (Italy)

Re: Comparing two effect sizes

Post by MAgojam »

Vit wrote:But I got a known MAJOR module error "Y is empty or has only NAs". This error happens when only 2 effects are entered. It works with >3 effects. This error should have been resolved in new versions (I am on Jamovi 2.3.0 and current MAJOR 1.2.1).
Hi Vit,
Jamovi updates are not necessarily reflected also on the modules available in the library.
Generally these modules are maintained by the authors themselves, who can respond to bug reports and fix them.
MAJOR is a fantastic module from Kyle Hamilton, but sadly Kyle hasn't been around for a while.
I'll ask Jonathon if he wants him to take a look at the MAJOR code, but tweaking other people's code isn't easy if you don't want to create more problems than you're trying to solve.
Cheers,
Maurizio
Post Reply