Maintaining Stability and Plasticity for Predictive Churn Reduction
George Adam Benjamin Haibe-Kains Anna Goldenberg
Deployed machine learning models should be updated to take advantage of a
larger sample size to improve performance, as more data is gathered over time.
Unfortunately, even when model updates improve aggregate metrics such as
accuracy, they can lead to errors on samples that were correctly predicted by
the previous model causing per-sample regression in performance known as
predictive churn. Such prediction flips erode user trust thereby reducing the
effectiveness of the human-AI team as a whole. We propose a solution called
Accumulated Model Combination (AMC) based keeping the previous and current
model version, and generating a meta-output using the prediction of the two
models. AMC is a general technique and we propose several instances of it, each
having their own advantages depending on the model and data properties. AMC
requires minimal additional computation and changes to training procedures. We
motivate the need for AMC by showing the difficulty of making a single model
consistent with its own predictions throughout training thereby revealing an
implicit stability-plasticity tradeoff when training a single model. We
demonstrate the effectiveness of AMC on a variety of modalities including
computer vision, text, and tabular datasets comparing against state-of-the-art
churn reduction methods, and showing superior churn reduction ability compared
to all existing methods while being more efficient than ensembles.