Fine tuning Pre trained Models for Robustness Under Noisy Labels
Sumyeong Ahn Sihyeon Kim Jongwoo Ko Se-Young Yun
The presence of noisy labels in a training dataset can significantly impact
the performance of machine learning models. To tackle this issue, researchers
have explored methods for Learning with Noisy Labels to identify clean samples
and reduce the influence of noisy labels. However, constraining the influence
of a certain portion of the training dataset can result in a reduction in
overall generalization performance. To alleviate this, recent studies have
considered the careful utilization of noisy labels by leveraging huge
computational resources. Therefore, the increasing training cost necessitates a
reevaluation of efficiency. In other areas of research, there has been a focus
on developing fine-tuning techniques for large pre-trained models that aim to
achieve both high generalization performance and efficiency. However, these
methods have mainly concentrated on clean datasets, and there has been limited
exploration of the noisy label scenario. In this research, our aim is to find
an appropriate way to fine-tune pre-trained models for noisy labeled datasets.
To achieve this goal, we investigate the characteristics of pre-trained models
when they encounter noisy datasets. Through empirical analysis, we introduce a
novel algorithm called TURN, which robustly and efficiently transfers the prior
knowledge of pre-trained models. The algorithm consists of two main steps: (1)
independently tuning the linear classifier to protect the feature extractor
from being distorted by noisy labels, and (2) reducing the noisy label ratio
and fine-tuning the entire model based on the noise-reduced dataset to adapt it
to the target dataset. The proposed algorithm has been extensively tested and
demonstrates efficient yet improved denoising performance on various benchmarks
compared to previous methods.