Characterizing and Measuring Linguistic Dataset Drift
Tyler A. Chang Kishaloy Halder Neha Anna John Yogarshi Vyas Yassine Benajiba Miguel Ballesteros Dan Roth
NLP models often degrade in performance when real world data distributions
differ markedly from training data. However, existing dataset drift metrics in
NLP have generally not considered specific dimensions of linguistic drift that
affect model performance, and they have not been validated in their ability to
predict model performance at the individual example level, where such metrics
are often used in practice. In this paper, we propose three dimensions of
linguistic dataset drift: vocabulary, structural, and semantic drift. These
dimensions correspond to content word frequency divergences, syntactic
divergences, and meaning changes not captured by word frequencies (e.g. lexical
semantic change). We propose interpretable metrics for all three drift
dimensions, and we modify past performance prediction methods to predict model
performance at both the example and dataset level for English sentiment
classification and natural language inference. We find that our drift metrics
are more effective than previous metrics at predicting out-of-domain model
accuracies (mean 16.8% root mean square error decrease), particularly when
compared to popular fine-tuned embedding distances (mean 47.7% error decrease).
Fine-tuned embedding distances are much more effective at ranking individual
examples by expected performance, but decomposing into vocabulary, structural,
and semantic drift produces the best example rankings of all considered
model-agnostic drift metrics (mean 6.7% ROC AUC increase).