Root Causing Prediction Anomalies Using Explainable AI
Ramanathan Vishnampet Rajesh Shenoy Jianhui Chen Anuj Gupta
This paper presents a novel application of explainable AI (XAI) for
root-causing performance degradation in machine learning models that learn
continuously from user engagement data. In such systems a single feature
corruption can cause cascading feature, label and concept drifts. We have
successfully applied this technique to improve the reliability of models used
in personalized advertising. Performance degradation in such systems manifest
as prediction anomalies in the models. These models are typically trained
continuously using features that are produced by hundreds of real time data
processing pipelines or derived from other upstream models. A failure in any of
these pipelines or an instability in any of the upstream models can cause
feature corruption, causing the model's predicted output to deviate from the
actual output and the training data to become corrupted. The causal
relationship between the features and the predicted output is complex, and
root-causing is challenging due to the scale and dynamism of the system. We
demonstrate how temporal shifts in the global feature importance distribution
can effectively isolate the cause of a prediction anomaly, with better recall
than model-to-feature correlation methods. The technique appears to be
effective even when approximating the local feature importance using a simple
perturbation-based method, and aggregating over a few thousand examples. We
have found this technique to be a model-agnostic, cheap and effective way to
monitor complex data pipelines in production and have deployed a system for
continuously analyzing the global feature importance distribution of
continuously trained models.