Monitoring Machine Learning Models: Online Detection of Relevant
  Deviations
Florian Heinrichs
Machine learning models are essential tools in various domains, but their
performance can degrade over time due to changes in data distribution or other
factors. On one hand, detecting and addressing such degradations is crucial for
maintaining the models' reliability. On the other hand, given enough data, any
arbitrary small change of quality can be detected. As interventions, such as
model re-training or replacement, can be expensive, we argue that they should
only be carried out when changes exceed a given threshold. We propose a
sequential monitoring scheme to detect these relevant changes. The proposed
method reduces unnecessary alerts and overcomes the multiple testing problem by
accounting for temporal dependence of the measured model quality. Conditions
for consistency and specified asymptotic levels are provided. Empirical
validation using simulated and real data demonstrates the superiority of our
approach in detecting relevant changes in model quality compared to benchmark
methods. Our research contributes a practical solution for distinguishing
between minor fluctuations and meaningful degradations in machine learning
model performance, ensuring their reliability in dynamic environments.