Verifix: Post-Training Correction to Improve Label Noise Robustness with
  Verified Samples
Sangamesh Kodge Deepak Ravikumar Gobinda Saha Kaushik Roy
Label corruption, where training samples have incorrect labels, can
significantly degrade the performance of machine learning models. This
corruption often arises from non-expert labeling or adversarial attacks.
Acquiring large, perfectly labeled datasets is costly, and retraining large
models from scratch when a clean dataset becomes available is computationally
expensive. To address this challenge, we propose Post-Training Correction, a
new paradigm that adjusts model parameters after initial training to mitigate
label noise, eliminating the need for retraining. We introduce Verifix, a novel
Singular Value Decomposition (SVD) based algorithm that leverages a small,
verified dataset to correct the model weights using a single update. Verifix
uses SVD to estimate a Clean Activation Space and then projects the model's
weights onto this space to suppress activations corresponding to corrupted
data. We demonstrate Verifix's effectiveness on both synthetic and real-world
label noise. Experiments on the CIFAR dataset with 25% synthetic corruption
show 7.36% generalization improvements on average. Additionally, we observe
generalization improvements of up to 2.63% on naturally corrupted datasets like
WebVision1.0 and Clothing1M.