| Random-Effects Model (k = 8) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Estimate | se | Z | p | CI Lower Bound | CI Upper Bound | ||||||||
| Intercept | -0.931 | 0.480 | -1.94 | 0.052 | -1.871 | 0.010 | |||||||
| . | . | . | . | . | . | ||||||||
| Note. Tau² Estimator: Restricted Maximum-Likelihood | |||||||||||||
| [3] | |||||||||||||
| Heterogeneity Statistics | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Tau | Tau² | I² | H² | R² | df | Q | p | ||||||||
| 1.330 | 1.77 (SE= 0.9849 ) | 97.11% | 34.599 | . | 7.000 | 139.810 | < .001 | ||||||||
The analysis was carried out using the standardized mean difference as the outcome measure. A random-effects model was fitted to the data. The amount of heterogeneity (i.e., tau²), was estimated using the restricted maximum-likelihood estimator (Viechtbauer 2005). In addition to the estimate of tau², the Q-test for heterogeneity (Cochran 1954) and the I² statistic are reported. In case any amount of heterogeneity is detected (i.e., tau² > 0, regardless of the results of the Q-test), a prediction interval for the true outcomes is also provided. Studentized residuals and Cook's distances are used to examine whether studies may be outliers and/or influential in the context of the model. Studies with a studentized residual larger than the 100 x (1 - 0.05/(2 X k))th percentile of a standard normal distribution are considered potential outliers (i.e., using a Bonferroni correction with two-sided alpha = 0.05 for k studies included in the meta-analysis). Studies with a Cook's distance larger than the median plus six times the interquartile range of the Cook's distances are considered to be influential. The rank correlation test and the regression test, using the standard error of the observed outcomes as predictor, are used to check for funnel plot asymmetry. A total of k=8 studies were included in the analysis. The observed standardized mean differences ranged from -3.7016 to 0.1704, with the majority of estimates being negative (88%). The estimated average standardized mean difference based on the random-effects model was \hat{\mu} = -0.9308 (95% CI: -1.8714 to 0.0099). Therefore, the average outcome did not differ significantly from zero (z = -1.9392, p = 0.0525). According to the Q-test, the true outcomes appear to be heterogeneous (Q(7) = 139.8102, p < 0.0001, tau² = 1.7700, I² = 97.1097%). A 95% prediction interval for the true outcomes is given by -3.7028 to 1.8413. Hence, although the average outcome is estimated to be negative, in some studies the true outcome may in fact be positive. An examination of the studentized residuals revealed that one study (Krajovica) had a value larger than ± 2.7344 and may be a potential outlier in the context of this model. According to the Cook's distances, one study (Krajovica) could be considered to be overly influential. The regression test indicated funnel plot asymmetry (p = 0.0344) but not the rank correlation test (p = 0.1087).
| Selection Model Results | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Estimate | SE | p-value | CI Lower Bound | CI Upper Bound | |||||
| 0.00 | 0.000 | 0.00 | 0.000 | 0.000 | |||||
| Note. Error during optimization, select another model type | |||||||||
| [3] | |||||||||
| Publication Bias Assessment | |||||
|---|---|---|---|---|---|
| Test Name | value | p | |||
| Fail-Safe N | 212.000 | < .001 | |||
| Begg and Mazumdar Rank Correlation | -0.500 | 0.109 | |||
| Egger's Regression | -2.115 | 0.034 | |||
| Trim and Fill Number of Studies | 0.000 | . | |||
| Note. Fail-safe N Calculation Using the Rosenthal Approach | |||||
| Test of Excess Significance | Significant Findings | |||
|---|---|---|---|
| Observed Number of Significant Findings | 6 | ||
| Expected Number of Significant Findings | 8 | ||
| Observed Number / Expected Number | 0.816 | ||
| [4] | |||
| Test of Excess Significance | Estimated Power of Tests | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Min | Q1 | Median | Q3 | Max | |||||
| 0.888 | 0.899 | 0.920 | 0.930 | 0.961 | |||||
| Note. Estimated Power of Tests (based on theta = -0.9308) | |||||||||
| [4] | |||||||||
Test of Excess Significance: p = 0.9785 ( X^2 = NA, df = 1). Limit Estimate: NA (where p = 0.1)
| Publication bias test p-uniform | |||
|---|---|---|---|
| Test Statistic | p-value | ||
| 0.000 | 0.990 | ||
| Note. Error | |||
| Effect size estimation p-uniform | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Effect Size Estimate | CI Lower Bound | CI upper Bound | Z | p-value | Number of Significant Studies | ||||||
| 0.000 | 0.000 | 0.000 | 0.000 | 0.990 | -1.000 | ||||||
| Note. Error | |||||||||||
| Random-Effects Model (k = 8) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Estimate | se | Z | p | CI Lower Bound | CI Upper Bound | ||||||||
| Intercept | -0.374 | 0.232 | -1.61 | 0.107 | -0.829 | 0.081 | |||||||
| . | . | . | . | . | . | ||||||||
| Note. Tau² Estimator: Restricted Maximum-Likelihood | |||||||||||||
| [3] | |||||||||||||
| Heterogeneity Statistics | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Tau | Tau² | I² | H² | R² | df | Q | p | ||||||||
| 0.615 | 0.3781 (SE= 0.2302 ) | 89.69% | 9.703 | . | 7.000 | 49.823 | < .001 | ||||||||
The analysis was carried out using the standardized mean difference as the outcome measure. A random-effects model was fitted to the data. The amount of heterogeneity (i.e., tau²), was estimated using the restricted maximum-likelihood estimator (Viechtbauer 2005). In addition to the estimate of tau², the Q-test for heterogeneity (Cochran 1954) and the I² statistic are reported. In case any amount of heterogeneity is detected (i.e., tau² > 0, regardless of the results of the Q-test), a prediction interval for the true outcomes is also provided. Studentized residuals and Cook's distances are used to examine whether studies may be outliers and/or influential in the context of the model. Studies with a studentized residual larger than the 100 x (1 - 0.05/(2 X k))th percentile of a standard normal distribution are considered potential outliers (i.e., using a Bonferroni correction with two-sided alpha = 0.05 for k studies included in the meta-analysis). Studies with a Cook's distance larger than the median plus six times the interquartile range of the Cook's distances are considered to be influential. The rank correlation test and the regression test, using the standard error of the observed outcomes as predictor, are used to check for funnel plot asymmetry. A total of k=8 studies were included in the analysis. The observed standardized mean differences ranged from -2.0275 to 0.0193, with the majority of estimates being negative (88%). The estimated average standardized mean difference based on the random-effects model was \hat{\mu} = -0.3738 (95% CI: -0.8288 to 0.0812). Therefore, the average outcome did not differ significantly from zero (z = -1.6103, p = 0.1073). According to the Q-test, the true outcomes appear to be heterogeneous (Q(7) = 49.8227, p < 0.0001, tau² = 0.3781, I² = 89.6939%). A 95% prediction interval for the true outcomes is given by -1.6619 to 0.9143. Hence, although the average outcome is estimated to be negative, in some studies the true outcome may in fact be positive. An examination of the studentized residuals revealed that one study (Krajovica) had a value larger than ± 2.7344 and may be a potential outlier in the context of this model. According to the Cook's distances, one study (Krajovica) could be considered to be overly influential. Neither the rank correlation nor the regression test indicated any funnel plot asymmetry (p = 0.5484 and p = 0.6008, respectively).
| Selection Model Results | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Estimate | SE | p-value | CI Lower Bound | CI Upper Bound | |||||
| 0.00 | 0.000 | 0.00 | 0.000 | 0.000 | |||||
| Note. Error during optimization, select another model type | |||||||||
| [3] | |||||||||
| Publication Bias Assessment | |||||
|---|---|---|---|---|---|
| Test Name | value | p | |||
| Fail-Safe N | 54.000 | < .001 | |||
| Begg and Mazumdar Rank Correlation | -0.214 | 0.548 | |||
| Egger's Regression | -0.523 | 0.601 | |||
| Trim and Fill Number of Studies | 0.000 | . | |||
| Note. Fail-safe N Calculation Using the Rosenthal Approach | |||||
| Test of Excess Significance | Significant Findings | |||
|---|---|---|---|
| Observed Number of Significant Findings | 3 | ||
| Expected Number of Significant Findings | 8 | ||
| Observed Number / Expected Number | 0.444 | ||
| [4] | |||
| Test of Excess Significance | Estimated Power of Tests | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Min | Q1 | Median | Q3 | Max | |||||
| 0.786 | 0.819 | 0.844 | 0.869 | 0.905 | |||||
| Note. Estimated Power of Tests (based on theta = -0.3738) | |||||||||
| [4] | |||||||||
Test of Excess Significance: p = 0.9997 ( X^2 = NA, df = 1). Limit Estimate: NA (where p = 0.1)
| Publication bias test p-uniform | |||
|---|---|---|---|
| Test Statistic | p-value | ||
| 0.000 | 0.990 | ||
| Note. Error | |||
| Effect size estimation p-uniform | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Effect Size Estimate | CI Lower Bound | CI upper Bound | Z | p-value | Number of Significant Studies | ||||||
| 0.000 | 0.000 | 0.000 | 0.000 | 0.990 | -1.000 | ||||||
| Note. Error | |||||||||||
[1] The jamovi project (2021). jamovi. (Version 2.2) [Computer Software]. Retrieved from https://www.jamovi.org.
[2] R Core Team (2021). R: A Language and environment for statistical computing. (Version 4.0) [Computer software]. Retrieved from https://cran.r-project.org. (R packages retrieved from MRAN snapshot 2021-04-01).
[3] Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software. link, 36, 1-48.
[4] Francis, G. (2013). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology. link, 57, 153-169.
[5] Lakens, D. (2017). Equivalence tests: A practical primer for t-tests, correlations, and meta-analyses. Social Psychological and Personality Science. link, 1, 1-8.
[6] van Houwelingen, H. C., Zwinderman, K. H., Stijnen, T. (1993). A bivariate approach to meta-analysis. Statistics in Medicine. link, 12, 2273-2284.