Measuring the Stability of Process Outcome Predictions in Online Settings
This work addresses the problem of ensuring consistency and reliability in predictive models for business process monitoring, particularly in high-risk scenarios, but it is incremental as it focuses on evaluation rather than new prediction methods.
The paper tackles the challenge of evaluating the stability of predictive process monitoring models in online settings by proposing an evaluation framework with four performance meta-measures, and validation on artificial and real-world event logs shows that these measures facilitate model comparison and selection for different risk scenarios.
Predictive Process Monitoring aims to forecast the future progress of process instances using historical event data. As predictive process monitoring is increasingly applied in online settings to enable timely interventions, evaluating the performance of the underlying models becomes crucial for ensuring their consistency and reliability over time. This is especially important in high risk business scenarios where incorrect predictions may have severe consequences. However, predictive models are currently usually evaluated using a single, aggregated value or a time-series visualization, which makes it challenging to assess their performance and, specifically, their stability over time. This paper proposes an evaluation framework for assessing the stability of models for online predictive process monitoring. The framework introduces four performance meta-measures: the frequency of significant performance drops, the magnitude of such drops, the recovery rate, and the volatility of performance. To validate this framework, we applied it to two artificial and two real-world event logs. The results demonstrate that these meta-measures facilitate the comparison and selection of predictive models for different risk-taking scenarios. Such insights are of particular value to enhance decision-making in dynamic business environments.