Measuring the Stability of Process Outcome Predictions in Online
  Settings
Suhwan Lee Marco Comuzzi Xixi Lu Hajo A. Reijers
Predictive Process Monitoring aims to forecast the future progress of process
instances using historical event data. As predictive process monitoring is
increasingly applied in online settings to enable timely interventions,
evaluating the performance of the underlying models becomes crucial for
ensuring their consistency and reliability over time. This is especially
important in high risk business scenarios where incorrect predictions may have
severe consequences. However, predictive models are currently usually evaluated
using a single, aggregated value or a time-series visualization, which makes it
challenging to assess their performance and, specifically, their stability over
time. This paper proposes an evaluation framework for assessing the stability
of models for online predictive process monitoring. The framework introduces
four performance meta-measures: the frequency of significant performance drops,
the magnitude of such drops, the recovery rate, and the volatility of
performance. To validate this framework, we applied it to two artificial and
two real-world event logs. The results demonstrate that these meta-measures
facilitate the comparison and selection of predictive models for different
risk-taking scenarios. Such insights are of particular value to enhance
decision-making in dynamic business environments.