Request for adding to the T-Test module the effect sizes R² / Eta squared and probability of superiority
Posted: Fri Sep 19, 2025 9:02 am
I would like to request a couple of what I believe might be important additions to the present output of the Jamovi Independent Samples T-Test module.
At present, the resulting table presents the difference between two group means and, if requested, the standardized effect size Cohen’s d. What I think is seriously lacking here is another scale-free effect size, namely R² (or Eta squared, which is identical in this case). Considering that a measure of explained variance (R² / Eta squared) is routinely provided in the regression / ANOVA / GLM modules, why not also providing it in the Independent Samples T-Test module and saving Jamovi users the trouble of running multiple group comparison analyses to get all the relevant effect sizes?
A second addition to the effect sizes I would like to suggest is stochastic superiority (a.k.a. probability of superiority effect size measure / the Common Language Effect Size), which presents the probability that a randomly selected respondent from group A has a higher score on the outcome variable than a randomly selected respondent from group B. This alternative measure of group differences is easy to calculate / to program, can be based both on Cohen’s d and Mann-Whitney U and is in my opinion a very useful additional effect size measure (just try for once, using the raw measurement units or, even worse, using Cohen’s d, to generate a clear intuition of the size of an intervention effect with researchers or other professionals who are not statistical experts). For example, if a mean difference of 4.41 points on some non-intuitive scale translates into a Cohen’s d of 1.35, then the corresponding probability of superiority is 0.83. This means that if you would take many randomly selected pairs of respondents from group A and group B, then in 83% of those pairs the respondent from group A has a higher score than the respondent of group B. This probability provides another kind of insight into the degree the two groups differ on the outcome variable.
I think that these two changes would provide the Jamovi user with (much needed) additional insight into the (practical / clinical) importance of the observed difference between two groups.
Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17(2), 137–152.
McGraw, K. O., & Wong, S. P. (1992). A common language effect size statistic. Psychological Bulletin, 111(2), 361–365. https://doi.org/10.1037/0033-2909.111.2.361
Ruscio, J. (2008). A probability-based measure of effect size: Robustness to base rates and other factors. Psychological Methods, 13, 19–30.
Ruscio, J., & Gera, B. L. (2013). Generalizations and extensions of the probability of superiority effect size estimator. Multivariate Behavioral Research, 48(2), 208–219.
Ruscio, J., & Mullen, T. (2012). Confidence intervals for the probability of superiority effect size measure and the area under a receiver operating characteristic curve. Multivariate Behavioral Research, 47, 201–223.
Vargha, A., & Delaney, H. D. (2000). A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25,101–132.
At present, the resulting table presents the difference between two group means and, if requested, the standardized effect size Cohen’s d. What I think is seriously lacking here is another scale-free effect size, namely R² (or Eta squared, which is identical in this case). Considering that a measure of explained variance (R² / Eta squared) is routinely provided in the regression / ANOVA / GLM modules, why not also providing it in the Independent Samples T-Test module and saving Jamovi users the trouble of running multiple group comparison analyses to get all the relevant effect sizes?
A second addition to the effect sizes I would like to suggest is stochastic superiority (a.k.a. probability of superiority effect size measure / the Common Language Effect Size), which presents the probability that a randomly selected respondent from group A has a higher score on the outcome variable than a randomly selected respondent from group B. This alternative measure of group differences is easy to calculate / to program, can be based both on Cohen’s d and Mann-Whitney U and is in my opinion a very useful additional effect size measure (just try for once, using the raw measurement units or, even worse, using Cohen’s d, to generate a clear intuition of the size of an intervention effect with researchers or other professionals who are not statistical experts). For example, if a mean difference of 4.41 points on some non-intuitive scale translates into a Cohen’s d of 1.35, then the corresponding probability of superiority is 0.83. This means that if you would take many randomly selected pairs of respondents from group A and group B, then in 83% of those pairs the respondent from group A has a higher score than the respondent of group B. This probability provides another kind of insight into the degree the two groups differ on the outcome variable.
I think that these two changes would provide the Jamovi user with (much needed) additional insight into the (practical / clinical) importance of the observed difference between two groups.
Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17(2), 137–152.
McGraw, K. O., & Wong, S. P. (1992). A common language effect size statistic. Psychological Bulletin, 111(2), 361–365. https://doi.org/10.1037/0033-2909.111.2.361
Ruscio, J. (2008). A probability-based measure of effect size: Robustness to base rates and other factors. Psychological Methods, 13, 19–30.
Ruscio, J., & Gera, B. L. (2013). Generalizations and extensions of the probability of superiority effect size estimator. Multivariate Behavioral Research, 48(2), 208–219.
Ruscio, J., & Mullen, T. (2012). Confidence intervals for the probability of superiority effect size measure and the area under a receiver operating characteristic curve. Multivariate Behavioral Research, 47, 201–223.
Vargha, A., & Delaney, H. D. (2000). A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25,101–132.