Cost-Effective Retraining of Machine Learning Models
Ananth Mahadevan Michael Mathioudakis
It is important to retrain a machine learning (ML) model in order to maintain
its performance as the data changes over time. However, this can be costly as
it usually requires processing the entire dataset again. This creates a
trade-off between retraining too frequently, which leads to unnecessary
computing costs, and not retraining often enough, which results in stale and
inaccurate ML models. To address this challenge, we propose ML systems that
make automated and cost-effective decisions about when to retrain an ML model.
We aim to optimize the trade-off by considering the costs associated with each
decision. Our research focuses on determining whether to retrain or keep an
existing ML model based on various factors, including the data, the model, and
the predictive queries answered by the model. Our main contribution is a
Cost-Aware Retraining Algorithm called Cara, which optimizes the trade-off over
streams of data and queries. To evaluate the performance of Cara, we analyzed
synthetic datasets and demonstrated that Cara can adapt to different data
drifts and retraining costs while performing similarly to an optimal
retrospective algorithm. We also conducted experiments with real-world datasets
and showed that Cara achieves better accuracy than drift detection baselines
while making fewer retraining decisions, ultimately resulting in lower total
costs.