Benford’s Law: Using JASP to Test Whether a Data Set Occurred Naturally

November 30 - 2021

Source: Getsnoopy, CC BY-SA 4.0, via Wikimedia Commons.

In 1938 Frank Benford observed that in many natural occurring sets of numerical data (e.g., population numbers, death rates, stock prices) a leading “1” (30.1%) appears more frequently than a leading “2” (17.6%), which in turn appears more frequently than a leading “3” (12.5%) and so forth. The relative frequencies of these leading digits are known as Benford’s law and can be thought of as a gold standard for “naturalness” of a set or list of numbers. For instance, to detect possible data tampering (e.g., made-up monetary amounts) in a list of financial transactions Nigiri (2012) suggested testing the leading digits in that list against Benford’s law.

The heuristic explanation of this law is rooted in the fact that every counting job must start by counting the lower numbers (e.g., 1s, 10s, or 100s) before higher numbers (e.g., 9s, 90s, or 900s) are reached. Not all counting jobs keep on going to also include higher numbers. Hence, survivorship bias combined with the natural order of counting imply that the higher digits have decreased chances to appear compared to leading digits.

For naturally occurring data sets, this decrease in chance can be succinctly described by a logarithm. More precisely, Benford’s law states that the digit $d$ is expected to appear as a leading digit with chance $p_{d} = \log_{10}(1 + \tfrac{1}{d} )$ . A plugin of $d=4$ into the formula yields 9.7%, and the same can be done for all digits $d=1,2, \ldots, 9$ resulting in the following Table 0:

Table 0. Benford’s law for leading digits refers to the collection of nine chances in the rightmost column.

Leading digit $d$	Expected relative frequency of appearance
1	0.301
2	0.176
3	0.125
4	0.097
5	0.079
6	0.067
7	0.058
8	0.051
9	0.046

The column of chances shown in Table 0 can be thought of as the expected chances of each digit appearing as a leading digit. Provided with a data set, we can calculate the observed relative frequency with which the nine digits appear as leading digits by simply counting and dividing. If the data set occurred naturally, then only a small deviation between the nine observed relative frequencies and the nine expected relative frequencies is anticipated. On the other hand, if the data set is tampered with, then the deviation between the nine observed relative frequencies and the nine expected relative frequencies is anticipated to be large. A multinomial test, as described in a previous blog post can be used to decide when a deviation between the observed and expected relative frequencies is large enough to reject the null hypothesis that the data set occurred naturally, i.e., Benford’s law.

Data

Many data sets follow Benford’s law. In this example, we perform an analysis inspired by Collins (2017), who examined the total number of registered citizens in the world’s 217 countries and dependencies from 2011 through 2015 as reported by the World Bank Group’s World DataBank and tested these data against Benford’s law. The data set can be found here in csv format and here as a JASP file. According to Benford’s law, the relative frequencies of leading digits in the total number of registered citizens per country should follow the distribution in the null hypothesis, i.e., the relative frequencies shown in Table 0.

Interface

In JASP the analysis to test against Benford’s law can be found in the Audit module.
Once the variable pop2015 (the countries’ populations in 2015) is put into the Variable field two tables appear: A table with the chi-squared ( $\chi^{2}$ ) value of the multinomial test, and underneath it a table with the observed relative frequencies of the leading digits.

Results

Let’s focus on the second table. By default Table – Frequency table is selected, which shows the observed relative frequencies of the leading digits in the data alongside the expected chances under Benford’s law, see Table 2 below.

By clicking the option Plot – Observed vs. expected a figure is produced that plots the observed relative frequencies of the leading digits in the data set against the expected relative frequencies under Benford’s law.

Figure 1. The relative frequencies of the leading digits in the data set compared to the expected relative frequencies of leading digits under Benford’s law.

As can be seen from the output in the table below, the $\chi^{2}$ -value for the data is 3.152. With 8 degrees of freedom (9 relative frequencies minus 1), the p-value is 0.92446 and the null hypothesis that the first digits in the data set are distributed according to Benford’s law cannot be rejected.

In addition to the p-value, JASP also reports a Bayes factor, which compares the null hypothesis that the data set occurred naturally (following Benford’s law) against the hypothesis that it is not, that is,

$H_{0}: p_{d} = \log_{10}(1 + \tfrac{1}{d}) \text{ vs } H_{1}: p_{d} \neq \log_{10}(1 + \tfrac{1}{d}) \text{ for } d=1,2, \ldots, 8, 9$

In the same way a multinomial $\chi^{2}$ -test was done for the p-value, JASP performs a Bayesian multinomial test for the Bayes factor, which is based on a uniform prior that is also referred to as $\text{Dirichlet}(\alpha_{1}, \ldots, \alpha_{9})$ with all $\alpha$ s equal to 1.

One advantage of the Bayes factor over the p-value is that it allows for the quantification of evidence in favor of the null –a non-significant p-value only indicates that the null cannot be rejected, but it cannot quantify evidence for the null. By default, JASP shows the Bayes factor in favor of the alternative over the null hypothesis $\text{BF}_{10}$ . For the problem at hand, the results are easier interpretable if the evidence is quantified for the null over the alternative instead, which can be requested by clicking Bayes factor – $\text{BF}_{01}$ . This leads to $\text{BF}_{01}=370,131.421$ which indicates that the data are about 370,131 times more likely under the null hypothesis that the data set follows Benford’s law than under the alternative that it did not.

Table 1. The results of the frequentist and Bayesian hypothesis tests.

Conclusion

For this data set the p-value analysis was non-significant, which implies that the null hypothesis that the leading digits follow Benford’s law cannot be rejected. The Bayesian result indicates that it is many times more likely that the leading digits in the data set follow Benford’s law than that they do not. Since both procedures come to similar conclusions, it seems like a safe bet to conclude that the leading digits of the total number of registered citizens in the world’s 217 countries follow Benford’s law.

Note that in this example we have only tested the first leading digits of a data set against Benford’s law. JASP also offers the possibility to test a set of numbers against Benford’s law for the first two digits, the last digit, and to test against the uniform distribution instead. We will leave that to the triggered reader to try out for themselves.

References

Collins, J. C. (2017). Using Excel and Benford’s Law to Detect Fraud. Retrieved on 15-03-2021 from https://www.journalofaccountancy.com/issues/2017/apr/excel-and-benfords-law-to-detect-fraud.html

Nigrini, M. J. (2012). Benford’s Law: Applications for forensic accounting, auditing, and fraud detection. John Wiley & Sons.

Sarafoglou, A., Aust, F., Marsman, M., Wagenmakers, E.-J., & Haaf, J.M. (2021). multibridge: An R package to evaluate informed hypotheses in binomial and multinomial models. Manuscript submitted for publication. https://psyarxiv.com/qk4cy

About the authors

Alexander Ly

Alexander Ly is the CTO of JASP and responsible for guiding JASP’s scientific and technological strategy as well as the development of some Bayesian tests.

Koen Derks

Koen Derks is a PhD candidate at Nyenrode Business University and at the Psychological Methods group at the University of Amsterdam. At JASP, he is creating JfA, an add-on module for Auditing.

The JASP Blog