Accounting for Publication Bias with Robust Bayesian Meta-Analysis in JASP

JASP 0.14 brings robust Bayesian meta-analysis (RoBMA). This extension of Bayesian meta-analysis allows researchers to adjust for publication bias when conducting model-averaged meta-analysis. RoBMA applies a set of twelve models simultaneously, some assuming publication bias and some assuming no publication bias. The inference will then be based most strongly on the models that predicted the data best. In other words, researchers don’t need to worry about which models to choose but the data will guide the inference to be based most strongly on the best models!

Example: Interracial dyads and performance

To see how this works in practice let’s take a look at the example reporting quoted from Bartos, Maier, & Wagenmakers (2020):

“Toosi et al. (2012) conducted a meta-analysis on the effect of “interracial” interactions on positive attitudes, negative affect, nonverbal behavior, and “objective measures of performance” (Toosi et al., 2012, p.1). The meta-analysis compared dyadic same-race versus “interracial” interactions. A standard reanalysis confirmed that “performance” was slightly better in dyads of the same-race compared to dyads of different race, r = 0.070, 95% CI [0.023, 0.117], p = .004, 𝜏 (on Cohen’s d scale) = 0.289, 95% CI [0.173, 0.370]. Toosi et al. (2012) applied Egger’s regression and reported a lack of funnel plot asymmetry, suggesting that the data set is not contaminated by publication bias.

We re-analyzed the same data set using Robust Bayesian Meta-Analysis. Before the analysis, we decided to use the default prior settings (i.e., standard normal distribution on effect sizes, inverse gamma distribution with 𝛼 = 1 and 𝛽 = 0.15 on heterogeneity, two two-sided weight functions with cut-points at (0.05) and (0.05, 0.10) and parameters  𝛼  = (1, 1) and (1, 1, 1), and the default point priors on the null hypotheses). We set the prior hypothesis probability to 0.50 for the effect size, heterogeneity, and publication bias. The models were estimated using correlations and sample sizes with Cohen’s d effect size transformation. The results did not indicate evidence for either presence or absence of the effect, BF10 = 2.01, they indicated strong evidence for heterogeneity, BFrf = 108.52, and they indicated strong evidence for publication bias, BF = 280.74. The resulting model-averaged effect size estimate was r = 0.032, 95% CI [0.000, 0.083], with the heterogeneity estimate, (on Cohen’s d scale) = 0.175, 95% CI [0.078, 0.291]. The MCMC diagnostics were good, with all Rhat values below 1.01 and all ESS above 800.”

The JASP implementation of RoBMA also provides several useful visualizations such as prior and posterior plots and plots of conditional estimates.

Figure 1. Prior (grey) and posterior (black) plot. We can see that the posterior for the null hypothesis (arrow) only reduces slightly, indicating weak evidence against the null.

Figure 2. Conditional estimates of all models assuming the alternative hypothesis to be true. Models assuming publication bias (2, 3, 5, & 6) indicate a much smaller conditional effect size.

Custom Priors

While default priors are a good choice in many situations, RoBMA also allows researchers to use a variety of different prior distributions (even for the null hypothesis). Thus, RoBMA can test a perinull (i.e., a hypothesis tightly centered around zero but not exactly zero) as well as comparison to an informed alternative. This GIF shows how to select custom priors in JASP:

We reanalyzed the data from Toosi et al., (2012) with normal(0, 0.1) as effect size under the null hypothesis and normal(0.6, 0.2) as effect size under the alternative hypothesis. This changed the results considerably, indicating strong evidence for the null hypothesis BF10 = 28.57. The associated prior and posterior distributions are shown in Figure 3.

Figure 3. Prior (grey) and posterior (black) plot. We can see that most of the posterior mass is in the area of the prior corresponding to the null hypothesis (i.e., the left hill).

Conclusion:

This blog post provides only a preview of what is possible when fitting Robust Bayesian meta-analysis. For more guidance we refer to the tutorial videos, papers, and posts below:

Tutorial Videos

Tutorial with default priors

How to fit custom priors

Papers and posts related to RoBMA

Bartoš, F., Maier, M., & Wagenmakers, E.-J. (2020). Adjusting for publication bias in JASP – Selection models and robust Bayesian meta-analysis. Manuscript submitted for publication.

Maier, M., Bartoš, F., & Wagenmakers, E.-J. (2020). Robust Bayesian meta-analysis: Addressing publication bias with model-averaging. Manuscript submitted for publication.

Gronau, Q. F., Heck, D., Berkhout, S., Haaf, J. M., & Wagenmakers, E-J.. (2020). A primer on Bayesian model-averaged meta-analysis. Manuscript submitted for publication. 

Gronau, Q. F., Heck, D., Berkhout, S., Haaf, J. M., & Wagenmakers, E-J.. (2020). How to Conduct a Bayesian Model-Averaged Meta-Analysis in JASP.

References

Toosi, N. R., Babbitt, L. G., Ambady, N., & Sommers, S. R. (2012). Dyadic interracial interactions: A meta-analysis. Psychological Bulletin, 138(1), 1–27. https://doi.org/10.1037/a0025767

About the authors

Eric-Jan Wagenmakers

Eric-Jan (EJ) Wagenmakers is professor at the Psychological Methods Group at the University of Amsterdam. EJ guides the development of JASP.

František Bartoš

František Bartoš is a PhD candidate at the Psychological Methods Group of the University of Amsterdam.

Maximilian Maier