What should we do about having given "shotguns to toddlers"? (Linear Mixed-Effects)

[reason180]

There seems to be a growing consensus that the p values provided by linear mixed-effects models are often wrong (though people tend to word it more delicately than that). In particular, it appears that the p values may (but not necessarily) be inflated when the model includes random slopes but not random intercepts:

• https://www.researchgate.net/publicatio ... nteraction

• Oberauer, K. (2022). The importance of random slopes in mixed models for Bayesian hypothesis testing. Psychological Science, 33(4), 648–665. https://doi.org/10.1177/09567976211046884

• https://www.middleprofessor.com/files/a ... k/lmm.html

However, including random slopes does not always work because in such situations there may not be enough data for the model to converge on a solution or to provide reasonable statistical power.

It has been said that encouraging scientists to use linear mixed effects models is "like giving shotguns to toddlers." So I'm feeling like a gun-toting toddler who's 99% confident in the correctness of p values provided by ANOVAs (when the underlying assumptions have been met), but only about 70% confident in p values provided by linear mixed effects models.

Should I be more confident?

Cheers.

[mcfanda@gmail.com]

Extremely more confident. Mixed models are simple linear models with possibility to have random coefficients across clustering variables. P-values are just fine when the model is well-defined (see for instance https://link.springer.com/article/10.37 ... 21-01546-0)

I wouldn't say that there is growing consensus about p-values being wrong in mixed models. There is, and there has always been, consensus on the fact that if one does not include random coefficients that are potentially varying across clusters, the inferential tests associated with the corresponding fixed effects are anti-conservative (so they are significant "too often") (you find this in the same literature you mentioned, but there are many other sources that agree with this point. see for instance https://www.sciencedirect.com/science/a ... 6X12001180 )

However, this issue does not imply that one should always include random coefficients, so the fact that "including random slopes does not always works" is not a problem. The inflation of type I error occurs when a coefficient has variability across cluster levels but it is not allowed to vary, so it is not set as random effect in the model. On the other hand, if the coefficient does not show variability in the data, removing it from the random effects does not inflate the type I error. On the contrary, it may increase power without introducing biases. see https://www.sciencedirect.com/science/a ... 6X17300013

Thus, the issue with mixed models is not whether the p-values are wrong (they are not), but whether one is setting the model up in the correct way. Let's say that the users should know what they are doing. However, this is basically true for any statistical technique.

[reason180]

OK. Thanks. But I see "anti-conservative; significant too often" (i.e., more often than 5% when the null hypothesis is true) as just a different way of saying "wrong," since a truly correct p value would not be "significant too often" and instead would be significant 5% of the time when the null is true.

I now feel more confident (say, 90%) that the p values from LMMs that include all possible random coefficients are not "anti-conservative; significant too often" (some would say "not the 'w'-word").

[Whirly123]

A really minor point, but just in case others stumble on this thread they might be confused by a typo in the original question:

In particular, it appears that the p values may (but not necessarily) be inflated when the model includes random slopes but not random intercepts

I think you meant to say in the other way round:

"In particular, it appears that the p values may (but not necessarily) be inflated when the model includes random intercepts but not random slopes"

[reason180]

Yes. Thanks for the correction.