Sensitivity analysis for LMM

Discuss statistics related things
Post Reply
Vit
Posts: 65
Joined: Fri Apr 19, 2019 11:51 am

Sensitivity analysis for LMM

Post by Vit »

Hi all,
I am looking for help. We received reviews for an article where we ran LMMs. The weakness of the study was a small sample, N=21 participants. We ran rather simple LMMs with a (continuous) rating of stimuli as a dependent variable, two continuous predictors as covariates, and stimuli and rater IDs as random effects in Jamovi. The models (based on Fs and associated p values) were not stat significant, though with R2C around 0.5, R2M ~ 0.02. Some covariates had estimates' slopes with 95%CI non-including zero and we discussed them in our results section.

The reviewer wants us to run a post hoc power analysis for all models.

The analysis is rather exploratory, we are aware of post hoc power pitfalls and we do not consider running it.

Is there a way how to run sensitivity analysis you could recommend or even provide help on how to perform? None of us is versed in R.

We also came across a preprint Murayama, Kou, Satoshi Usami, and Michiko Sakaki. “Summary-Statistics-Based Power Analysis: A New and Practical Method to Determine Sample Size for Mixed-Effects Modelling.” Preprint. Open Science Framework, May 11, 2020. https://doi.org/10.31219/osf.io/6cer3.

To our understanding, they suggest that simple LMMs power calculations based on simple tests (e.g. t-test) available in GPower are a good approximation. We tried running sensitive for linear regression, for alpha 0.05, beta 0.8, N=21, 2 predictors and got f2=0.545 which is equal to R2=0.35. If compared to the observed R2C, we should have enough sensitivity for R2C we observed. This makes no sense, as our models were non-significant.


What would you suggest we should do or reply?
Vit
Posts: 65
Joined: Fri Apr 19, 2019 11:51 am

Re: Sensitivity analysis for LMM

Post by Vit »

Update:

so far I came across the simr R package that seems to do the job of listing observed power and allowing for simulations (extending sample with simulated data along or within chosen variable) and drawing power curves.
Post Reply