JASP 0.14 brings robust Bayesian meta-analysis (RoBMA). This extension of Bayesian meta-analysis allows researchers to adjust for publication bias when conducting model-averaged meta-analysis. RoBMA applies a set of twelve models simultaneously, some assuming publication bias and some assuming no publication bias. The inference will then be based most strongly on the models that predicted the data best. In other words, researchers don’t need to worry about which models to choose but the data will guide the inference to be based most strongly on the best models!
Example: Interracial dyads and performance
To see how this works in practice let’s take a look at the example reporting quoted from Bartos, Maier, & Wagenmakers (2020):
“Toosi et al. (2012) conducted a meta-analysis on the effect of “interracial” interactions on positive attitudes, negative affect, nonverbal behavior, and “objective measures of performance” (Toosi et al., 2012, p.1). The meta-analysis compared dyadic same-race versus “interracial” interactions. A standard reanalysis confirmed that “performance” was slightly better in dyads of the same-race compared to dyads of different race, r = 0.070, 95% CI [0.023, 0.117], p = .004, 𝜏 (on Cohen’s d scale) = 0.289, 95% CI [0.173, 0.370]. Toosi et al. (2012) applied Egger’s regression and reported a lack of funnel plot asymmetry, suggesting that the data set is not contaminated by publication bias.
We re-analyzed the same data set using Robust Bayesian Meta-Analysis. Before the analysis, we decided to use the default prior settings (i.e., standard normal distribution on effect sizes, inverse gamma distribution with 𝛼 = 1 and 𝛽 = 0.15 on heterogeneity, two two-sided weight functions with cut-points at (0.05) and (0.05, 0.10) and parameters 𝛼 = (1, 1) and (1, 1, 1), and the default point priors on the null hypotheses). We set the prior hypothesis probability to 0.50 for the effect size, heterogeneity, and publication bias. The models were estimated using correlations and sample sizes with Cohen’s d effect size transformation. The results did not indicate evidence for either presence or absence of the effect, BF10 = 2.01, they indicated strong evidence for heterogeneity, BFrf = 108.52, and they indicated strong evidence for publication bias, BF⍵⍵ = 280.74. The resulting model-averaged effect size estimate was r = 0.032, 95% CI [0.000, 0.083], with the heterogeneity estimate, (on Cohen’s d scale) = 0.175, 95% CI [0.078, 0.291]. The MCMC diagnostics were good, with all Rhat values below 1.01 and all ESS above 800.”
The JASP implementation of RoBMA also provides several useful visualizations such as prior and posterior plots and plots of conditional estimates.
Figure 1. Prior (grey) and posterior (black) plot. We can see that the posterior for the null hypothesis (arrow) only reduces slightly, indicating weak evidence against the null.
Figure 2. Conditional estimates of all models assuming the alternative hypothesis to be true. Models assuming publication bias (2, 3, 5, & 6) indicate a much smaller conditional effect size.
Custom Priors
While default priors are a good choice in many situations, RoBMA also allows researchers to use a variety of different prior distributions (even for the null hypothesis). Thus, RoBMA can test a perinull (i.e., a hypothesis tightly centered around zero but not exactly zero) as well as comparison to an informed alternative. This GIF shows how to select custom priors in JASP:
We reanalyzed the data from Toosi et al., (2012) with normal(0, 0.1) as effect size under the null hypothesis and normal(0.6, 0.2) as effect size under the alternative hypothesis. This changed the results considerably, indicating strong evidence for the null hypothesis BF10 = 28.57. The associated prior and posterior distributions are shown in Figure 3.
Figure 3. Prior (grey) and posterior (black) plot. We can see that most of the posterior mass is in the area of the prior corresponding to the null hypothesis (i.e., the left hill).
Conclusion:
This blog post provides only a preview of what is possible when fitting Robust Bayesian meta-analysis. For more guidance we refer to the tutorial videos, papers, and posts below:
Tutorial Videos
Papers and posts related to RoBMA
References
Toosi, N. R., Babbitt, L. G., Ambady, N., & Sommers, S. R. (2012). Dyadic interracial interactions: A meta-analysis. Psychological Bulletin, 138(1), 1–27. https://doi.org/10.1037/a0025767
About the authors
