Source: shutterstock.com/szczygiel
The analyses in this blog post are based on an example from Bartoš et al. (2021).
Publication bias is a serious problem for meta-analyses that can lead to inflated effect size estimates. In order to adjust for and assess the effect of publication bias –a preferential publishing of statistically significant studies (Rosenthal & Gaito, 1964)– on the meta-analytic effect size estimate, we proceed with one of the many publication bias adjustment techniques: PET-PEESE. PET-PEESE adjusts for the linear or quadratic relationship between effect sizes and standard errors that are assumed to be unrelated under no publication bias (Stanley & Doucouliagos, 2014; Stanley, 2017). We do that with the newly added PET-PEESE analysis added to the Meta-Analysis module (see this blogpost for a synopsis of our new pre-print on publication bias adjustment with JASP & R).
Data
We perform a meta-analysis of the correlation of acculturation mismatch (i.e., the contrast between the collectivist cultures of Asian and Latin immigrant groups and the individualistic culture of the United States) with the intergenerational cultural conflict (Lui, 2015). The data set can be found here in csv format and here as a JASP file. Lui (2015) collected 18 independent studies correlating acculturation mismatch with intergenerational cultural conflict. In the original article, Lui (2015) performed a standard random-effects reanalysis that found a statistically significant positive relationship between acculturation mismatch and the intergenerational cultural conflict, r = 0.250, p < .001.
Interface
To adjust for publication bias with PET-PEESE, we switch the data input to Correlations & N and pass appropriate dataset columns to the Correlation and N variable forms.
Results
The first output we see is the Model Tests output section with the Test of Effect and the Test of Publication Bias tables. Both tables are based on the PET model which is correctly specified under the absence of the effect. As we can see, the test for the presence of the effect under PET is not statistically significant; therefore, we interpret the mean effect size estimate based on PET, ⍴ = 0.000, 95% CI [-0.207, 0.205], from the Mean Estimates table in the Estimates output section. We would have interpreted the effect size estimate based on PEESE if we rejected the hypothesis of no effect at 𝛼 = 0.10 (Stanley & Doucouliagos, 2014; Stanley, 2017).
We can further visualize the estimated PET regression line fitted to the individual studies’ effect size estimates and standard errors by checking the PET checkbox under the Meta-Regression Estimate heading in the Plots section. The figure shows that smaller studies (with larger standard errors) produce inflated estimates. The adjusted mean effect size estimate then corresponds to the intersection with the y-axis, i.e., the predicted effect size estimate of a hypothetical study with no standard error.
References
Bartoš, F., Maier, M., Quintana D. S., & Wagenmakers, E. (2021). Adjusting for publication bias in JASP & R — Selection models, PET-PEESE, and robust Bayesian meta-analysis, Preprint. https://doi.org/10.31234/osf.io/75bqn
Lui, P. P. (2015). Intergenerational cultural conflict, mental health, and educational outcomes among Asian and Latino/a Americans: Qualitative and meta-analytic review. Psychological Bulletin, 141, 404. https://doi.org/10.1037/a0038449
Stanley, T. D., & Doucouliagos, H. (2014). Meta‐regression approximations to reduce publication selection bias. Research Synthesis Methods, 5, 60-78. https://doi.org/10.1002/jrsm.1095
Stanley, T. D. (2017). Limitations of PET-PEESE and other meta-analysis methods. Social Psychological and Personality Science, 8, 581-591. https://doi.org/10.1177/1948550617693062
Rosenthal, R., & Gaito, J. (1964). Further evidence for the cliff effect in interpretation of levels of significance. Psychological Reports, 15, 570. https://doi.org/10.2466/pr0.1964.15.2.570
About the authors
