I noticed strange results with a Bayesian RM ANOVA with two factors (2 and 11 levels, respectively). The attachment shows the results, compared with those obtained with JASP, and the are major discrepancies. It looks like something is not working properly in the JSQ module in Jamovi.
Hi, I'm attaching the .omv file (zipped). The Bayesian ANOVA is at the bottom of the Results section. IRRPRO should have a BF10 around 2 or so, but it's 10^7 ...
ah, wait, i remember this one. the jasp guys changed the RM ANOVA implementation. we use the RM ANOVA implementation from the BayesFactor R package. the two implementations are different, and i take the fact that BayesFactor isn't updating it's implementation as a sign that they think the original procedure is sound.
so i think this might be different stats folks disagreeing on which procedure is the best for a given design (which is fairly standard).
Yes, it may be fairly standard that "different stats folks [disagree] on which procedure is the best for a given design," but while there is strong consensus about how to implement a classic t test, ANOVA, or regression (you tend to get exactly the same results regardless of which stats package you use), this is not the case for Bayesian analysis. In my opinion, the reason is that the application of Bayesian analysis to non-binomial data and vague-ish hypotheses is somewhat forced, leading to lots of complexity, and lots of freedom to shop around for an implementation that happens to provide a desirable results. For this reason, while I somewhat-trust Bayesian versions of *simple* classical analyses such as t test, my trust breaks down for more complicated analyses such as ANOVAs.
Last edited by reason180 on Thu May 11, 2023 4:54 pm, edited 1 time in total.
Very helpful, and something that I hadn't heard much about, not being a Bayesian analysis expert. I had no idea that different methods could lead to such a wild range of results: from a BF=2 (consistent with the standard NHT ANOVA results) to a BF>10^7
Is the moral that we just shouldn't rely on Bayesian analyses for RM ANOVAs (at least, until there is some consensus)?
