The new Predictive Analytics module introduces extensive time series prediction methods to JASP. It integrates classical time series techniques and machine learning models with quality control concepts.
Quality control involves keeping processes within predefined boundaries to maintain the desired quality. By applying time series forecasting, we can predict how a process will evolve over time, allowing us to anticipate and prevent that a process goes out of control in the future. Since the prediction models in the module are probabilistic, they not only forecast the moment when and if a process might go out of control, but also quantify the uncertainty associated with these predictions. Furthermore, the JASP module uses ensemble Bayesian Model Averaging to select and combine the best prediction models based on their historical performance.
We will first explain the basic concepts that the module draws upon in more detail and then provide a tutorial on how to use the new JASP module in practice.
Quality Control
In quality control, we usually have a process that continuously generates data and we want to monitor it over time to make sure that it stays within the desired boundaries over time. For example, in a healthcare setting, we might monitor employee absenteeism rates to detect potential disease outbreaks (Woodall & Montgomery, 2014). In manufacturing, we might track the dimensions of products to ensure they meet quality standards before they are sold.
Probalistic Forecasting
Sometimes, we’re not just interested in whether a process is currently out of control but also whether it is going to be out-of-control in the future. Time series prediction allows us to forecast future values based on past data. This is especially valuable in production processes with multiple steps: If we only detect faulty products at the end of the production cycle, it can lead to increased losses, as all parts in the current cycle might need to be discarded. By predicting whether a process is going to go out of control, we can act in advance, saving time and money.
Forecast Evaluation
However, in order to make decisions based on our predictions we need to have confidence in its accuracy. One common way to evaluate the forecast accuracy is to repeatedly subset available historical data into a training and test data sets, which is also called cross-validation. The training set is used to train the statistical models which then make predictions about the future values in the test set that are then compared to the real observation, allowing us to quantify how well the models predict unobserved observations. We can compute various forecasting metrics that quantify the accuracy of each model and help with determining the best model. The accuracy is assessed by forecast metrics that are based on point predictions (such as R-squared or mean absolute error) as well as a probabilistic metrics that take the account the whole predictive distribution (e.g. Logarithmic score or the Continuous Ranked Probability Score).
Ensemble Bayesian Model Averaging
Instead of relying on the single best model to predict the future, we can also combine the predictions of different models. First, this has the benefit that it takes into account the uncertainty we have in our model selection as simply selecting a single best model might produce overconfident results (Wagenmakers et al., 2022). Additionally, the problem at hand might be too complex to be represented well by a single model – combining different models might provide a better predictuon if the if each model capture a unique aspect of the problem at hand (Sagi & Rokach, 2018). As a last point, it reduces the chances of selecting the wrong model by the user as they don’t have to manually pick the one they consider the best.
Tutorial: Simulated Data Set
The module is quite extensive but we can break down the process into three steps:
- Determining the control bounds
- Selecting and evaluating the candidate models
- Predicting the future with the best model
The basic idea is to first set the control limits, select various possible models and test their predictive accuracy on past data and finally use these results to select the final model that we want to use to predict the future. By using past performance as a proxy for future prediction performance we should be able to make a more informed choice of which model will perform the best.
We will demonstrate the module’s capabilities using a simulated dataset representing a real-world production process. Our dependent variable is the size of produced goods, measured at the end of the production cycle. Since one production cycle for a piece takes approximately 100 data points from start to finish, our goal is to predict 100 data points into the future. This allows us to intervene if the next cycle is predicted to go out of control. We also have measurements from various sensors (e.g., temperature, vibration, machine part age) taken at the start of the cycle, which are somewhat related to the process. To predict into the future, we use the sensor measurements at the beginning of the cycle, lagged by 100 data points.
The process values are primarily influenced by the machinery’s heat, which depends on whether the machine is left on overnight or how long it’s turned off for maintenance. Operating daily from 09:00 to 17:00 and producing 60 pieces per minute, the dataset represents seven days of data.
Determining the Control Bounds
Selecting necessary variables
First, we need to load the data into JASP (see Figure 1).
Figure 1. Selecting necessary variables
Our Dependent Variable (y) is the continuous process we want to monitor and predict. The Time variable (time) contains the timestamps. While supplying Covariates and Factors is optional, we include them because our sensor measurements are associated with the dependent variable and could improve predictive accuracy. Variables like day_new_time_since and day_new_n_since indicate how much time has passed since the day’s start and how many pieces have been produced, respectively. These are useful because the machine’s operating time directly affects its temperature, influencing the size of the produced pieces. The Include in Training variable indicates whether a data point should be used for training and verifying the model (set to 1) or for predicting the future (set to 0). For future predictions, while the dependent variable’s observations are unavailable, supplying covariates and factors is necessary if included in our models. While the observations of the dependent variable are of course not available for these data points as we want to predict them, it is needed to supply covariates and factors if we included them in our models.
Control plot and control limits
After supplying the variables, JASP automatically generates a basic control plot:
The red dashed lines indicate the control bounds we aim not to cross. By default, these bounds are set so that any data more than two standard deviations from the mean is flagged as out-of-control.
Visual inspection reveals two key observations. First, there’s a cyclic pattern where the process abruptly increases in size and then gradually decreases. This relates to temperature effects: the machine cools down overnight when turned off, causing produced pieces to be larger when restarted. As the machine runs continuously, it heats up, and the piece size stabilizes.
Second, the process appears to go out of control rapidly at the end of the time series, possibly due to a machine part malfunction causing excessive heat and reducing product size. Since exceeding control bounds is undesirable, we’ll test whether the module can accurately predict past instances when the process went out of control—a practice known as forecast verification. This ensures we can trust our models before using them to predict the unseen future.
Before proceeding, let’s adjust the control limits. In production, acceptable size ranges are often known, so we don’t need to base control bounds solely on statistical deviations. We’ll increase the control limits slightly since only the last part of the time series clearly goes out of control. This adjustment is made under the Time Series Descriptives section:
Under the Error Bound Selection option we can select Manual Bounds and then manually input the upper/lower limit of 8.2/7.15 which changes the plot into the following:
Selecting and Evaluating the Candidate Models
Choosing an evaluation plan
As mentioned previously, we want to choose the model that predicts the future the best. One method for this is called cross validation where the existing data is randomly split into training and test data sets. The training set is used to train the statistical model which then make predictions about the test set that are compared to the real observation. By comparing the predictions with the true observations we can test how well each model predicts the unseen data and generalizes.
As the goal is to evaluate the predictive performance on unseen future observation, normal cross validation that randomly selects training and test data is not appropriate. Instead, the data is split in such a way that the training data is always temporally before the test data. In our example we want to train the models on some amount of past data and then use the next 100 data points as test data as this constitutes one production cycle. Afterwards we shift the training window ahead by a certain amount of data points, we retrain our models and then predict the next 100 data points. This is called rolling window cross validation or historical forecast evaluation.
The settings for this procedure can be found under the “Forecast Evaluation” section in the JASP module:
Figure 5: Creating an evaluation plan
The Training window option refers to how many past data points our model is trained on. Since we experience an underlying shift during the last day of the process and it goes out of control, we will only train the models on the past 75 data points. In practice it might be unknown on how much data each model should be trained on and different options should be considered. If the underlying data generating process stays constant over time, the models are ideally trained on all of the past – this can be achieved by checking the option for Cumulative Training. If the process changes (quickly) over time — as indicated by worse predictive performance a shorter training window might be a better choice.
The Prediction window option determines how many data points should be predicted into the future. This is ideally chosen in such a way that it corresponds to the prediction horizon one is interested in predicting (as different models might perform better in short and other in long-term forecasts). In our example, this corresponds to a prediction horizon of 100 as this represents one full cycle.
The next option called Skip between training slices indicates how many data points are shifted ahead for each prediction slice.
The Maximum nr. of slices option determines how many folds are created. Increasing this number leads to less prediction slices or folds that are computed but might increase computation time. Especially if computational time is scarce since we want to make a timely prediction about the future, we might want to select a smaller number. In our case we selected 6 as we want to only predict the last part of the process where the data goes out-of-control.
When we check the Show evaluation option we can also see the evaluation plan visualized to check which data the model will evaluate (see Figure 6). Here we see that the blue colored data is the training period that the models are trained on- whereas the red period will be predicted and compared with the actual observations.
Selecting candidate models
Now that we have a plan to evaluate the data, we can select the models that we want to use to predict the future. We can find the option for selecting different models under the Forecast Evaluation section under Model Choice. On the left side of the selection option, all available models are displayed while the right side indicates the models we have chosen for prediction.
The naming convention for the models is as follows: The model family is indicated as the first part of the model name before the hyphen “-”, and the text afterwards indicates the model specification.
Figure 7: Displays all available prediction models
For example, in our selected models, the prophet model is a Bayesian time series model that automatically models appropriate seasonality via Fourier orders, trends, and change points. As it is a time series model, it predicts future observations of our dependent variable based on its own past values. If we wanted to add covariates, we would select the prophet – regression model. Our second model, denoted as bart — regression, is a Bayesian Additive Regression Tree model. It combines multiple decision trees into a large ensemble model and is able to model high-dimensional non-linear relationships. As it is not a time series model by default, we need to supply it at least some covariates, which in our case are the sensors. Additionally, it is also possible to supply it with our lagged dependent variable. If we enable this option in the Feature Engineering section, internally a new column is created that contains our dependent variable but at a previous time point. That way, even models that are not time series models can model the relationship of a variable with itself over time. When the model then predicts future data points in our test set, the predicted value for time point t is then iteratively used as an input covariate for the prediction of time point t+1. Note that this takes a long time for the BART model, as the model is computationally complex.
After we have selected which models we would like to use for prediction, the models are trained and evaluated based on the evaluation plan we specified earlier. After the computations are finished, we get a table with the corresponding evaluation metrics and are also able to plot our predictions.
Choosing an appropriate evaluation metric
One can evaluate the prediction accuracy of our models by computing so-called evaluation or forecast metrics that measure how well our models predict the future by comparing the actual observations of the test set with the predictions. The JASP module offers both probabilistic and point-based prediction metrics: Instead of relying on the point or mean prediction of a model, probabilistic forecast metrics compute how much the observed value differs from the whole predictive distribution. That way, we are able to take into account the uncertainty that is associated with our predictive distribution. Ideally, our distribution is very sharp, as this indicates that we are very certain in our prediction.
For the current example, we focus on the mean absolute error (MAE), the Continuous Ranked Probability Score (CRPS), and the R-squared value. As the name suggests, the MAE is calculated by computing how each observation differs from the forecast in absolute terms (i.e., removing the minus of negative numbers) and averaging it across all predicted observations. The CRPS, on the other hand, generalizes the MAE as it computes it for the whole predictive distribution instead of only a single forecast value for each observation. The R-squared value tells us how much variance in the real observation is explained by the forecast. In general, all forecast evaluation metrics can be selected under the section Forecast Evaluation -> Evaluation Metrics:
Forecast Evaluation Metric Table
Let’s examine the table with forecast evaluation metrics. The first row indicates the model name. The other rows show the values of the corresponding evaluation metrics, averaged across all evaluation slices. Lower MAE and CRPS values indicate better performance, while higher R-squared values indicate better predictions. While MAE and CRPS depend on the scale of our data (e.g., if we multiplied all observations by 10, the MAE values would also increase by 10), R-squared values are scale-independent. A value of R-squared = 1 indicates perfect prediction by our model, while R-squared = 0 indicates that our predictions fail completely. By comparing the values for each model, we can determine which candidate model performs best.
Comparing the prophet and BART models, we see that the prophet model generally predicts better, as both the CRPS and MAE values are lower. On the other hand, the R-squared value for the BART model is higher, which can be explained by its greater flexibility. While it explains more variance in the observed data, this extra flexibility leads to larger errors on average, as measured by the MAE and CRPS. This will become clearer when we examine the data plots.
Forecast Plots
We can also visualize the predictions of each model against the real observations under the section Prediction Plots.
The corresponding prediction plot then appears in the JASP output. As shown in the legend, different line colors represent different model types. The real observed data is shown in purple, the prophet model in green, and the BART model in yellow. The plot displays both the training window used to train all models and the prediction horizon used to calculate the evaluation metrics. The two dashed red lines depict the control limits, allowing us to verify whether the models accurately predict when the data goes out of control. Each evaluation slice is shown as a separate plot.
Generally, we can see that both models predict the data quite well. The process only goes significantly out of control in the last two slices around data point 2500, which is more evident in the overall control plot in Figure 4. The prophet model seems better at predicting the overall trend of the data, while the BART model captures the fluctuations better but does not pick up the downward trend as effectively. This makes sense since the prophet model is designed to predict trends and seasonality in time series data, while the BART model excels at modeling relationships with covariates and does not explicitly model the process’s relationship with itself. This aligns with the evaluation metric table: the prophet model had lower errors as measured by MAE and CRPS, while the BART model explained more variance in the real observations. Overall, we can conclude that we do a decent job predicting when the process goes out of control.
Selecting the best model for future prediction
After evaluating the predictive performance of our models on past data, we are ready to select a model from our candidate pool to predict the future. From the metric table and prediction plot, we’ve observed that the prophet model excels at predicting the overall trend, while the BART model is better at capturing variance in the data. Ideally, we would have a single model proficient in both trend forecasting and modeling complex covariate relationships. When this isn’t feasible due to computational or statistical limitations, we can instead combine predictions from different models. This approach eliminates the need for users to manually choose between models—which can benefit applied users—and potentially improves predictive accuracy over any single model. One method to combine different model predictions—and which is implemented in this module — is called Ensemble Bayesian Model Averaging.
Ensemble Bayesian Model Averaging
Ensemble Bayesian Model Averaging (eBMA) consists of two steps: First, model weights are assigned to all models based on their past predictive performance. These weights, called posterior model probabilities, can be interpreted as the relative likelihood that the true observations originated from each evaluated model compared to all others. Second, these weights are used to weight the individual model predictions for future data, which are then summed to create a single ensemble model. In the JASP module, weights are estimated separately for each evaluation slice and then used to adjust the predictions for the next slice. We then compute evaluation metrics for the eBMA predictions to determine the predictive benefit compared to any single model. Finally, we use the model weights from the last evaluation slice to adjust out-of-sample predictions for the future.
To use eBMA in JASP, we simply check the Perform eBMA box in the Ensemble Bayesian Model Averaging section. A prerequisite is that we have performed forecast evaluation with at least two models. The Method option determines how the model weights are calculated and is set by default to the faster Expectation-Maximization algorithm. The Evaluation Method determines how the predictive benefit of eBMA is assessed. Additionally, we can display the model weights averaged across all evaluation slices—or per individual slice if we want to know specifically what weights were used.
As a result, we obtain a table with the model weights and an adjusted evaluation metric table that includes the eBMA model. Across all slices, the prophet model receives a slightly higher model weight, indicating it is more likely to be the best model. This aligns with the evaluation table, where the prophet model slightly outperformed the BART model in terms of MAE and CRPS.
Examining the evaluation metrics, we see that the eBMA predictions decrease the CRPS values by approximately 17% and the MAE values by about 15% compared to the prophet model. The R-squared value increased by 12.5% with the eBMA approach. Based on these metrics, we can argue that combining predictions via eBMA has substantially improved prediction accuracy.
But how do these 15–20% improvements translate in practice? In our example dataset, we want to predict whether a process goes out of control. Is the increased prediction accuracy enough to make better decisions about whether we should turn the machine off or intervene ahead of time? To explore this, we can look at the prediction plots we discussed earlier. We can now visualize the eBMA predictions. As an example, we’ll investigate two prediction slices. In both cases, the predictions of the prophet model (blue) are slightly too low, and the predictions of the BART model (green) are somewhat too high. The eBMA predictions, however, are better overall and closer to the real observations shown in purple.
Predicting the Future
So far, we’ve set the control limits, evaluated all models, and explored whether the eBMA approach can help us make better predictions. Now it’s time to predict the actual future. Let’s pretend a new day has started, and we want to test whether the module can predict if the process will go out of control. We’ve also fixed our production machine and replaced some parts to prevent it from going out of control again. Let’s assume the database connected to our module has been updated with new data. The process has not gone out of control, and the corresponding control plot looks like this:
To predict the future, we open the Future Prediction section. The first option lets us select the model we want to use for prediction. Here, we choose the Ensemble Bayesian Model Averaging, as it automatically weights all selected models for us and has shown predictive benefits over a single model. The model weights are taken from the most recent training to adjust future predictions.
The values for the prediction horizon (how far we want to predict into the future) and the training window (how much past data we want to use for training) are by default set to the same values we selected for model evaluation. Alternatively, we could set custom values or train the models on all past data. The Future prediction plot checkbox is automatically enabled as soon as a model is chosen for prediction.
Finally, we examine the prediction plot that appears after computations are complete. The horizontal red dashed lines represent the control bounds. The data depicted in black before the vertical dashed line represents real data, while the data afterward shows the predicted future. The blue area depicts the 95% credible interval, which can be interpreted as follows: Given all past data and the predictions of the different models, the eBMA method predicts that the unknown future observations will fall within this interval with a probability of 95%. Since neither the black line (mean prediction) nor the credible interval falls outside the control limits, we can assume that the process will not go out of control within the next 100 data points.
Summary
In this tutorial, we provided an overview of how to use the new Predictive Analytics module with a simulated dataset. We showed how to evaluate the predictive performance of various statistical models—including linear regression, classical time series models, and machine learning algorithms—using both point-based and probabilistic prediction metrics. We also demonstrated how to use Ensemble Bayesian Model Averaging to combine predictions from different models to obtain better predictions than any single model. Finally, we showed how to use these model predictions to forecast whether a process will go out of control.
Additional Features
This tutorial did not cover all the functionality of the module, as that would have been too extensive. A few notable examples are:
- Time Series Diagnostics: The module offers several diagnostic tools such as Autocorrelation Function Plots, which enable us to determine how related our dependent variable is with itself and thus assess its suitability for time series models. You can also create outlier tables to identify specific observations that went out of control.
- Feature Engineering: Some models, such as BART, cannot model autoregressive or seasonal components by default. Therefore, we added the ability to manually create lagged versions of the dependent variable to enable iterative predictions and add automatic time-variable features, allowing these models to accurately model seasonality.
About The Author
Fridtjof Petersen
Fridtjof Petersen is a PhD student at the University of Groningen.
References
Sagi, O., & Rokach, L. (2018). Ensemble learning: A survey. WIREs Data Mining and Knowledge Discovery, 8(4). https://doi.org/10.1002/widm.1249
Wagenmakers, E.-J., Sarafoglou, A., & Aczel, B. (2022). One statistical analysis must not rule them all. Nature,605(7910), 423–425. https://doi.org/10.1038/d41586-022-01332-8
Woodall, W. H., & Montgomery, D. C. (2014). Some current directions in the theory and application of statistical process monitoring. Journal of Quality Technology, 46(1), 78–94. https://doi.org/10.1080/00224065.2014.11917955
About the author
