The goal of this JASP blog post is threefold:
- To demonstrate the improved ability to annotate analyses. For annotations, JASP 0.12 now uses Quill. As stated on https://quilljs.com/, “Quill is a free, open source WYSIWYG editor built for the modern web. With its modular architecture and expressive API, it is completely customizable to fit any need.“
- To demonstrate the ease with which JASP output can be transformed to a blog post. Just go to the hamburger menu (leftmost icon on the ribbon) and then select “Export Results”. After selecting a file location, all of the output –annotations, tables, and figures– will be saved as an html file.
- To showcase how a Bayesian meta-analyses, new in JASP 0.12, can quantify evidence in favor of the absence of an effect. Here we focus on a series of 6 studies recently reported by Kristal et al. (2020; PNAS). The abstract reads:
Honest reporting is essential for society to function well. However, people frequently lie when asked to provide information, such as misrepresenting their income to save money on taxes. A landmark finding published in PNAS [L. L. Shu, N. Mazar, F. Gino, D. Ariely, M. H. Bazerman, Proc. Natl. Acad. Sci. U.S.A. 109, 15197–15200 (2012)] provided evidence for a simple way of encouraging honest reporting: asking people to sign a veracity statement at the beginning instead of at the end of a self-report form. Since this finding was published, various government agencies have adopted this practice. However, in this project, we failed to replicate this result. Across five conceptual replications (n = 4,559) and one highly powered, preregistered, direct replication (n = 1,235) conducted with the authors of the original paper, we observed no effect of signing first on honest reporting. Given the policy applications of this result, it is important to update the scientific record regarding the veracity of these results.
Below we conduct a Bayesian meta-analysis (for a primer see Gronau et al., 2020; for an application to the Many Labs 4 project, see Haaf et al., 2020) that includes the six replication studies. Note that Kristal et al. did use Bayes factor hypothesis tests to quantify evidence in favor of the absence of an effect (!), but applied these tests to each study separately.
A First Look
Before we really get going, let’s execute some basic analyses first. In the main input panel, we change the “Bayes Factor” from BF10 to BF01, as we expect to see evidence in favor of the null hypothesis. For now we won’t change the settings in the submenu “Prior”, so JASP will use its default settings there. In the submenu “Plots”, we go to “Forest plot” and tick “Both” to see both observed and estimated effect sizes.
Let’s look at the output:
Posterior Estimates per Model | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 95% Credible Interval | |||||||||||||
| Mean | SD | Lower | Upper | BF₀₁ | |||||||||
| Fixed effects | μ | 0.018 | 0.027 | -0.033 | 0.070 | 26.802 | |||||||
| Random effects | μ | 0.021 | 0.041 | -0.060 | 0.106 | 19.545 | |||||||
| τ | 0.068 | 0.034 | 0.026 | 0.158 | 10.709 | ᵃ | |||||||
| Averaged | μ | ᵇ | 0.019 | 0.026 | -0.031 | 0.066 | 26.182 | ||||||
| τ | ᶜ | 14.492 | |||||||||||
| Note. μ and τ are the group-level effect size and standard deviation, respectively. | |||||||||||||
| ᵃ Bayes factor of the fixed effects H₁ over the random effects H₁. | |||||||||||||
| ᵇ Model averaged posterior estimates are based on the models that assume an effect to be present. The Bayes factor is based on all four models: fixed effects H₀ & random effects H₀ over the fixed effects H₁ & random effects H₁. | |||||||||||||
| ᶜ Model averaged posterior estimates for τ are not yet available, but will be added in the future. The Bayes factor is based on all four models: fixed effects H₀ & H₁ over the random effects H₀ & H₁. | |||||||||||||
This table shows the following:
- For both the fixed-effect and random-effects models, the point estimate for the group mean mu is about 0.02 — very slightly positive, but in the direction opposite to that expected from the 2012 studies.
- The Bayes factor in favor of H0 is 26.80 for the fixed-effect model, 19.54 for the random-effects model, and 26.18 in case model-averaging is applied.
So: the data show some solid evidence for the absence of an effect. Let’s consider the forest plot:
Forest Plot
Observed and estimated study effects
The forest plot shows how the estimated effects are shrunken toward the group mean. The narrow intervals for Studies 5 and 6 confirm that these studies had large sample sizes. In four out of the six studies, the observed effect size is in the direction opposite to that expected.
In JASP, it is also possible (with one tickmark) to see a cumulative forest plot, and to see the prior and posterior distributions for the model parameters. In this post we focus on the evidence for the null hypothesis, and how this may change as a consequence of specifying a directional version of the alternative hypothesis.
A Second Look: A Directional Prior
For this analysis, we stick to the default prior on the group mean effect size mu, but we now incorporate the knowledge that the hypothesis of interest stipulates that the effect size ought to be negative rather than positive. This means that under the heading “Prior”, we tick the box that says “Upper bound: 0”.
Let’s look at the output:
Posterior Estimates per Model | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 95% Credible Interval | |||||||||||||
| Mean | SD | Lower | Upper | BF₀₁ | |||||||||
| Fixed effects | μ | -0.016 | 0.013 | -0.050 | -7.472e -4 | 52.770 | |||||||
| Random effects | μ | -0.027 | 0.025 | -0.091 | -6.961e -4 | 31.716 | |||||||
| τ | 0.072 | 0.039 | 0.026 | 0.168 | 8.826 | ᵃ | |||||||
| Averaged | μ | ᵇ | -0.019 | 0.016 | -0.059 | -8.832e -4 | 50.627 | ||||||
| τ | ᶜ | 14.506 | |||||||||||
| Note. μ and τ are the group-level effect size and standard deviation, respectively. | |||||||||||||
| ᵃ Bayes factor of the fixed effects H₁ over the random effects H₁. | |||||||||||||
| ᵇ Model averaged posterior estimates are based on the models that assume an effect to be present. The Bayes factor is based on all four models: fixed effects H₀ & random effects H₀ over the fixed effects H₁ & random effects H₁. | |||||||||||||
| ᶜ Model averaged posterior estimates for τ are not yet available, but will be added in the future. The Bayes factor is based on all four models: fixed effects H₀ & H₁ over the random effects H₀ & H₁. | |||||||||||||
This table shows that the evidence in favor of the null hypothesis has increased: it is now 52.77 for the fixed-effect model (up from 26.80), 31.72 for the random-effects model (up from 19.54), and 50.63 in case model-averaging is applied (up from 26.18).
The reason why the evidence against the alternative hypothesis has increased is that the directional predictions conflict much more with the data than the undirectional predictions. In other words, the negative-effects-only alternative hypothesis predicts the (slightly) positive effects from the data worse than the less risky “both positive and negative effects are possible” alternative hypothesis. Of course, there is a strong argument to be made that the directional hypothesis is the one we care about, so that the current analysis addresses a more pertinent question than the previous analysis featuring an undirectional alternative hypothesis.
Let’s do a sequential analysis and see how the evidence changes as the studies come in. We go to “Plots” and tick the options under “Sequential plot”:
Sequential Analysis
The first plot below shows the model-averaged Bayes factor as the studies accumulate. After three studies, the evidence in favor of the null hypothesis is already relatively high. The second plot below shows the model-averaged evidential flow for the presence of heterogeneity. To get a better impression of what models predict well and what models predict poorly, we also tick “Posterior model probabilities” under “Sequential plots”. As the third figure below demonstrates, the fixed-effects null hypothesis outpredicts the other three models.
We have only scratched the surface here of what additional insights a Bayesian meta-analysis can offer. Importantly, these insights are easy to obtain — placing a few tick marks is all it takes. We just put two papers on PsyArXiv that provide more background and guidance:
Gronau, Q. F., Heck, D. W., Berkhout, S. W., Haaf, J. M., & Wagenmakers, E.-J. (2020). A primer on Bayesian model-averaged meta-analysis. Manuscript submittted for publication. https://psyarxiv.com/97qup
Haaf, J., Hoogeveen, S., Berkhout, S., Gronau, Q. F., & Wagenmakers, E.-J. (2020). A Bayesian multiverse analysis of Many Labs 4: Quantifying the evidence against mortality salience. https://psyarxiv.com/cb9er/
Bayes factors effect size
Bayes factors heterogeneity
Posterior model probabilities
References
Gronau, Q. F., Heck, D. W., Berkhout, S. W., Haaf, J. M., & Wagenmakers, E.-J. (2020). A primer on Bayesian model-averaged meta-analysis. Manuscript submitted for publication. https://psyarxiv.com/97qup
Haaf, J., Hoogeveen, S., Berkhout, S., Gronau, Q. F., & Wagenmakers, E.-J. (2020). A Bayesian multiverse analysis of Many Labs 4: Quantifying the evidence against mortality salience. https://psyarxiv.com/cb9er/
Kristal, A. S., Whillans, A. V., Bazerman, M. H., Gino, F., Shu, L. L., Mazar, N., & Ariely, D. (2020). Signing at the beginning versus at the end does not decrease dishonesty. Proceedings of the National Academy of Sciences of the United States of America, 117, 7103-7107.
Like this post?
Subscribe to our newsletter to receive regular updates about JASP including our latest blog posts, JASP articles, example analyses, new features, interviews with team members, and more! You can unsubscribe at any time.About the authors
About the authors
