Tackling Noisy Labels with Network Parameter Additive Decomposition
Jingyi Wang Xiaobo Xia Long Lan Xinghao Wu Jun Yu Wenjing Yang Bo Han Tongliang Liu
Given data with noisy labels, over-parameterized deep networks suffer
overfitting mislabeled data, resulting in poor generalization. The memorization
effect of deep networks shows that although the networks have the ability to
memorize all noisy data, they would first memorize clean training data, and
then gradually memorize mislabeled training data. A simple and effective method
that exploits the memorization effect to combat noisy labels is early stopping.
However, early stopping cannot distinguish the memorization of clean data and
mislabeled data, resulting in the network still inevitably overfitting
mislabeled data in the early training stage.In this paper, to decouple the
memorization of clean data and mislabeled data, and further reduce the side
effect of mislabeled data, we perform additive decomposition on network
parameters. Namely, all parameters are additively decomposed into two groups,
i.e., parameters $\mathbf{w}$ are decomposed as
$\mathbf{w}=\bm{\sigma}+\bm{\gamma}$. Afterward, the parameters $\bm{\sigma}$
are considered to memorize clean data, while the parameters $\bm{\gamma}$ are
considered to memorize mislabeled data. Benefiting from the memorization
effect, the updates of the parameters $\bm{\sigma}$ are encouraged to fully
memorize clean data in early training, and then discouraged with the increase
of training epochs to reduce interference of mislabeled data. The updates of
the parameters $\bm{\gamma}$ are the opposite. In testing, only the parameters
$\bm{\sigma}$ are employed to enhance generalization. Extensive experiments on
both simulated and real-world benchmarks confirm the superior performance of
our method.