The only thing worse than statistical software that does not work, is statistical software that appears to work but produces incorrect output. For this reason, verifying the computed results of statistical software is serious business – also for JASP. This is why in July of 2020 we started a systematic verification process. Beginning with the most commonly used frequentist analyses, standard data sets have been analyzed both with JASP and with four other statistical software packages: SPSS (https://www.ibm.com/products/spss-statistics), SAS (https://www.sas.com/en_us/home.html), Minitab (https://www.minitab.com/en-us/), and R (https://www.r-project.org/). Whenever feasible, the computations were also checked by hand. The documentation of the output and more details of the process are now publicly accessible under https://jasp-stats.github.io/jasp-verification-project.
The structure of the verification website follows the structure of the modules in JASP. For example, if you are looking for a verification of the Binomial Test, you will find it – just as in JASP – under the header Frequencies.
Figure 1: JASP Screenshot Figure 2: Website Menu Screenshot
The verification of the Binomial Test then contains screenshots of the output generated by applying the test to the same dataset in different statistical software (i.e., JASP, SPSS, SAS, Minitab, and R). The example below shows JASP output followed by SPSS output.
Figure 3: Website Binomial Test Screenshot
To ensure transparency we have shared both the datasets and the code used for the computations in the different software programs. The code-snippets can be found in the documentation, right above the related output. The datasets are available on the GitHub page of the project, in the Datasets folder, https://github.com/jasp-stats/jasp-verification-project/tree/main/Datasets.
Up until now, about half of the – steadily increasing – number of options available in JASP have been verified in this manner. As expected (since JASP uses R under the hood) the results show that virtually all the verified output matched with the other statistical software packages and hand calculations. The options that remain to be verified are mainly Bayesian tests, whose verification will require a different approach, due to their absence in most other statistical software packages. However, as this process is an important step to ensure the quality of JASP, we aim to systematically verify these remaining options also as soon as possible.
About the authors
