Reliable and Interpretable Drift Detection in Streams of Short Texts
Ella Rabinovich Matan Vetzler Samuel Ackerman Ateret Anaby-Tavor
Data drift is the change in model input data that is one of the key factors
leading to machine learning models performance degradation over time.
Monitoring drift helps detecting these issues and preventing their harmful
consequences. Meaningful drift interpretation is a fundamental step towards
effective re-training of the model. In this study we propose an end-to-end
framework for reliable model-agnostic change-point detection and interpretation
in large task-oriented dialog systems, proven effective in multiple customer
deployments. We evaluate our approach and demonstrate its benefits with a novel
variant of intent classification training dataset, simulating customer requests
to a dialog system. We make the data publicly available.