Cross-Validation Is All You Need: A Statistical Approach To Label Noise
  Estimation
Jianan Chen Anne Martel
Label noise is prevalent in machine learning datasets. It is crucial to
identify and remove label noise because models trained on noisy data can have
substantially reduced accuracy and generalizability. Most existing label noise
detection approaches are designed for classification tasks, and data cleaning
for outcome prediction analysis is relatively unexplored. Inspired by the
fluctuations in performance across different folds in cross-validation, we
propose Repeated Cross-Validations for label noise estimation (ReCoV) to
address this gap. ReCoV constructs a noise histogram that ranks the noise level
of samples based on a large number of cross-validations by recording sample IDs
in each worst-performing fold. We further propose three approaches for
identifying noisy samples based on noise histograms to address increasingly
complex noise distributions. We show that ReCoV outperforms state-of-the-art
algorithms for label cleaning in a classification task benchmark. More
importantly, we show that removing ReCoV-identified noisy samples in two
medical imaging outcome prediction datasets significantly improves model
performance on test sets. As a statistical approach that does not rely on
hyperparameters, noise distributions, or model structures, ReCoV is compatible
with any machine learning analysis.