A Gradient-based Approach for Online Robust Deep Neural Network Training
  with Noisy Labels
Yifan Yang Alec Koppel Zheng Zhang
Learning with noisy labels is an important topic for scalable training in
many real-world scenarios. However, few previous research considers this
problem in the online setting, where the arrival of data is streaming. In this
paper, we propose a novel gradient-based approach to enable the detection of
noisy labels for the online learning of model parameters, named Online
Gradient-based Robust Selection (OGRS). In contrast to the previous sample
selection approach for the offline training that requires the estimation of a
clean ratio of the dataset before each epoch of training, OGRS can
automatically select clean samples by steps of gradient update from datasets
with varying clean ratios without changing the parameter setting. During the
training process, the OGRS method selects clean samples at each iteration and
feeds the selected sample to incrementally update the model parameters. We
provide a detailed theoretical analysis to demonstrate data selection process
is converging to the low-loss region of the sample space, by introducing and
proving the sub-linear local Lagrangian regret of the non-convex constrained
optimization problem. Experimental results show that it outperforms
state-of-the-art methods in different settings.