We wish to reanalyze the BGC vaccine dataset that was used as evidence of a positive preventive effect of the BGC vaccine against tuberculosis. As in our previous blog post about PET-PEESE, we want to assess the effect of publication bias –a preferential publishing of statistically significant studies (Rosenthal & Gaito, 1964)– on the meta-analytic effect size estimate. In contrast to PET-PEESE, WAAP-WLS specifies a different set of assumptions about the published set of studies and attempts to mitigate the publication bias by only considering highly powered studies (Ioannidis, Stanley, & Doucouliagos, 2017; Stanley et al., 2017). We do that with the newly added WAAP-WLS analysis in the Meta-Analysis module.
WAAP-WLS is a meta-analytic estimator that specifies a WAAP (weighted average of adequately powered studies, Ioannidis, Stanley, & Doucouliagos, 2017; Stanley et al., 2017) and WLS (weighted least squares, Stanley & Doucouliagos, 2017) models. A difference of the WAAP-WLS in contrast to random-effects models (traditionally used in the field of psychology) is that it does not assume additive heterogeneity. This difference should make it less susceptible to model misspecification and lead to results that are almost as efficient as when additive heterogeneity holds (Stanley & Doucouliagos, 2017). Furthermore, the WAAP method takes into account only studies with sufficient power (> 80%) to detect the meta-analytic effect size, reducing the influence of small-study effects. When there are no sufficiently powered studies, the WLS model using all estimates can be used.
Interface
We load the BGC vaccine data included in the JASP Data Library under the Meta-Analysis section and we open the WAAP-WLS analysis from the Meta-Analysis section. We use the ES column to specify the Effect Size and the SE column to specify the Effect Size Standard Error. The JASP file can be found here.
Results
The Test of Effect table summarizes the tests for the presence of the effect based on WLS and WAAP. As we can see in the footnote below the table, only 4 studies achieved sufficient power to detect the WLS effect size estimate. As a consequence, we cannot reject the null hypothesis of no effect of BGC vaccine with WAAP.
We further select the Mean model estimates checkbox under the Plots section to visualize the resulting WLS and WAAP estimates. We can see that even though the preventive effect is not statistically significant with the WAAP estimator, the effect size estimate is in line with the WLS estimate. Therefore, the lack of statistical significance seems to be due to the increased uncertainty in the effect size estimate when excluding the insufficiently powered studies rather than small-study effects.
These results are more in line with the fixed-effects meta-analytic estimate -0.436, 95% CI [-0.519, -0.353], than the random-effects estimate -0.745, 95% CI [-1.110, -0.381] (both of which can be obtained with the frequentist meta-analysis).
References
IIoannidis, J., Stanley, T. D., & Doucouliagos, H. (2017). The power of bias in economics research. The Economic Journal, 127, F236-F265. https://doi.org/10.1111/ecoj.12461
Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. (2017). Finding the power to reduce publication bias. Statistics in Medicine, 36, 1580-1598. https://doi.org/10.1002/sim.7228
Stanley, T. D., & Doucouliagos, H. (2017). Neither fixed nor random: Weighted least squares meta‐regression. Research Synthesis Methods, 8, 19-42. https://doi.org/10.1002/jrsm.1211
Rosenthal, R., & Gaito, J. (1964). Further evidence for the cliff effect in interpretation of levels of significance. Psychological Reports, 15, 570. https://doi.org/10.2466/pr0.1964.15.2.570
About the authors
