Out-of-Distribution Detection and Data Drift Monitoring using
  Statistical Process Control
Ghada Zamzmi Kesavan Venkatesh Brandon Nelson Smriti Prathapan Paul H. Yi Berkman Sahiner Jana G. Delfino
Background: Machine learning (ML) methods often fail with data that deviates
from their training distribution. This is a significant concern for ML-enabled
devices in clinical settings, where data drift may cause unexpected performance
that jeopardizes patient safety.
  Method: We propose a ML-enabled Statistical Process Control (SPC) framework
for out-of-distribution (OOD) detection and drift monitoring. SPC is
advantageous as it visually and statistically highlights deviations from the
expected distribution. To demonstrate the utility of the proposed framework for
monitoring data drift in radiological images, we investigated different design
choices, including methods for extracting feature representations, drift
quantification, and SPC parameter selection.
  Results: We demonstrate the effectiveness of our framework for two tasks: 1)
differentiating axial vs. non-axial computed tomography (CT) images and 2)
separating chest x-ray (CXR) from other modalities. For both tasks, we achieved
high accuracy in detecting OOD inputs, with 0.913 in CT and 0.995 in CXR, and
sensitivity of 0.980 in CT and 0.984 in CXR. Our framework was also adept at
monitoring data streams and identifying the time a drift occurred. In a
simulation with 100 daily CXR cases, we detected a drift in OOD input
percentage from 0-1% to 3-5% within two days, maintaining a low false-positive
rate. Through additional experimental results, we demonstrate the framework's
data-agnostic nature and independence from the underlying model's structure.
  Conclusion: We propose a framework for OOD detection and drift monitoring
that is agnostic to data, modality, and model. The framework is customizable
and can be adapted for specific applications.