TRIAGE: Characterizing and auditing training data for improved
  regression
Nabeel Seedat Jonathan Crabbé Zhaozhi Qian Mihaela van der Schaar
Data quality is crucial for robust machine learning algorithms, with the
recent interest in data-centric AI emphasizing the importance of training data
characterization. However, current data characterization methods are largely
focused on classification settings, with regression settings largely
understudied. To address this, we introduce TRIAGE, a novel data
characterization framework tailored to regression tasks and compatible with a
broad class of regressors. TRIAGE utilizes conformal predictive distributions
to provide a model-agnostic scoring method, the TRIAGE score. We operationalize
the score to analyze individual samples' training dynamics and characterize
samples as under-, over-, or well-estimated by the model. We show that TRIAGE's
characterization is consistent and highlight its utility to improve performance
via data sculpting/filtering, in multiple regression settings. Additionally,
beyond sample level, we show TRIAGE enables new approaches to dataset selection
and feature acquisition. Overall, TRIAGE highlights the value unlocked by data
characterization in real-world regression applications