An Empirical Study of Uncertainty Estimation Techniques for Detecting
  Drift in Data Streams
Anton Winter Nicolas Jourdan Tristan Wirth Volker Knauthe Arjan Kuijper
In safety-critical domains such as autonomous driving and medical diagnosis,
the reliability of machine learning models is crucial. One significant
challenge to reliability is concept drift, which can cause model deterioration
over time. Traditionally, drift detectors rely on true labels, which are often
scarce and costly. This study conducts a comprehensive empirical evaluation of
using uncertainty values as substitutes for error rates in detecting drifts,
aiming to alleviate the reliance on labeled post-deployment data. We examine
five uncertainty estimation methods in conjunction with the ADWIN detector
across seven real-world datasets. Our results reveal that while the SWAG method
exhibits superior calibration, the overall accuracy in detecting drifts is not
notably impacted by the choice of uncertainty estimation method, with even the
most basic method demonstrating competitive performance. These findings offer
valuable insights into the practical applicability of uncertainty-based drift
detection in real-world, safety-critical applications.