New: A Tutorial Paper on Binary Classification with JASP

Abstract

Medical professionals, patients, students, and the public at large regularly need to interpret the outcome of medical tests. These tests are error-prone, however, and the fact that the outcome is positive (or negative) does not establish with certainty that the disease is present (or absent). The correct interpretation of the test outcome demands that Bayes’ rule is used to combine the quality of the test (i.e., sensitivity and specificity) with available background information (i.e., disease prevalence). It is well known that people find it difficult to understand and apply Bayes’ rule, and that the correct outcome is often at odds with intuition, especially when a test with good operating characteristics yields a positive result, but the disease is relatively rare. Less well known is that in practical application, the values of sensitivity, specificity, and prevalence are usually associated with considerable uncertainty. The correct interpretation of a medical test demands that this uncertainty is explicitly acknowledged and properly taken into account.

To facilitate the correct interpretation of fallible medical tests we introduce the Binary classification module in the open-source software JASP. This module explains medical testing  through a series of informative visualizations. The module also allows users to propagate uncertainty in sensitivity, specificity, and prevalence to derived measure of interest, such as positive predictive value. The module can be used both in teaching and in medical practice.

Teaser

We are pleased to announce a new preprint titled: “Correct conclusions from fallible medical tests: A tutorial with JASP”, accessible at https://osf.io/jksz6. The tutorial exemplifies the use of a Binary classification analysis in JASP for the case of Binarius’ malaria scare:

Binarius, a 23-year old student at NYU, experiences fatigue and headache. Binarius fears having contracted malaria after a recent visit to Central Africa. They decide to take a rapid diagnostic test (RDT) for malaria and it comes back positive. What is the probability that Binarius has the disease? In technical terms, what is the positive predictive value?

We equip ourselves with JASP, empirical study of the performance of RDT for malaria (Berzosa et al., 2018), and with our wits of course, to give Binarius the best-informed answer regarding their medical status.

The tutorial is split in two parts. The aim of the first part of the tutorial is to expose users to the fundamentals of Bayes’ theorem through its application in binary classification settings. Through a series of visualizations we build up an intuition for the correct reasoning. Example visualizations, such as the icon plot and the area plot, are displayed below. The module provides more plots that go to even more depth, connecting the example to signal detection theory and discussing tools such as ROC curves.


Figure 1. Output from JASP. The icon plot displays a hypothetical population of 100 people that either have or do not have the disease, and test either negative or positive. Given that prevalence is relatively low (90 out of 100 people do not have the disease) and specificity is relatively high, most of the population would be true negative; here, about 80 in 100 (the blue icons). However, even with a relatively high specificity, we nevertheless obtain a number of false positive cases; here, about 10 in 100 (the yellow icons). This number outweighs the number of true positives in the population; here, about 8 in 100 (the green icons), and so even after Binarius tested positive it is somewhat more likely that they are false positive than true positive.


Figure 2. Output from JASP. The area plot displays the proportions of true positives, false positives, false negatives, and true negatives as rectangles on a unit square. The advantage of this layout is that changing either prevalence, sensitivity, or specificity alters only a single aspect of the rectangles, which facilitates a better understanding of how the three parameters affect the interpretation of the test outcome. For example, increasing prevalence would increase the width of the green rectangle and decrease the width of the orange rectangle while keeping their heights constant, meaning that the proportion True positives increases and the proportion False positives decreases. Thus, increasing (decreasing) prevalence while keeping sensitivity and specificity constant can be easily seen to increase (decrease) the PPV. NB. The area plot was inspired by Sanderson (2019; see the 3Blue1Brown YouTube video here); to the best of our knowledge it has not yet been applied in educational practice, other than by ourselves.

The second part of the tutorial focuses on an aspect of binary classification problems that is often not mentioned in educational materials: the fact that sensitivity, specificity, and prevalence are unknown quantities and therefore often associated with considerable uncertainty. This uncertainty thus needs to be propagated through our calculations and so instead of a single number of the PPV, we end up with a range (or more precisely, a distribution) of possibilities, as shown in Figure 3 below. This part typifies the notion that learning Bayes needs not only understanding the rules of probability and the Bayes’ theorem, but also adopting the mindset of learning in the presence of uncertainty.


Figure 3. Output from JASP. Posterior distributions of prevalence, sensitivity, specificity, and positive predictive value. The entire distributions are plotted as density estimates, with the black dot corresponding to the mean, the thick black line to the 67% and the thin black line to the 95% central credible interval, respectively.

The tutorial is accompanied by two .jasp files (https://osf.io/kue5h/) that provide the reader with the analysis conducted with JASP. One of the files showcases the functionality provided by the Binary classification analysis in JASP. The other file is intended for disciples of Bayesian statistics who are interested in how to conduct the Bayesian analysis of the binary classification problem using the JAGS module.

We hope that the tutorial proves valuable to those who would like to understand Bayes’ theorem as a stepping stone to learn (Bayesian) statistics, but also to those whose aims are more pragmatic: To understand how to derive correct conclusions from fallible medical tests.

References

Berzosa, P., de Lucio, A., Romay-Barja, M., Herrador, Z., González, V., García, L., Fernández-Martínez, A., Santana-Morales, M., Ncogo, P., Valladares, B., Riloha, M., & Benito, A. (2018). Comparison of three diagnostic methods (microscopy, RDT, and PCR) for the detection of malaria parasites in representative samples from Equatorial Guinea. Malaria Journal, 17(1), 1–12.

Kucharský, Š., & Wagenmakers, E. (2023). Correct conclusions from fallible medical tests: A tutorial with JASP. OSF Preprints. doi: 10.31219/osf.io/jksz6

Sanderson, G. (2019). Bayes theorem, the geometry of changing beliefs.
3Blue1Brown channel on YouTube. Retrieved from https://youtu.be/HZGCoVF3YvM?si=eBkLa1hvYCtt53hm

About the author

Šimon Kucharský

Šimon is a PhD candidate at the Psychological Methods Group of the University of Amsterdam.