Deep convolutional neural networks (CNNs) are known to be robust against label noise on extensive datasets. However, at the same time, CNNs are capable of memorizing all labels even if they are random, which means they can memorize corrupted labels. Are CNNs robust or fragile to label noise? Much of researches focusing on such memorization uses class-independent label noise to simulate label corruption, but this setting is simple and unrealistic. In this paper, we investigate the behavior of CNNs under class-dependently simulated label noise, which is generated based on the conceptual distance between classes of a large dataset (i.e., ImageNet-1k). Contrary to previous knowledge, we reveal CNNs are more robust to such class-dependent label noise than class-independent label noise. We also demonstrate the networks under class-dependent noise situations learn similar representation to the no noise situation, compared to class-independent noise situations. Deep convolutional neural networks (CNNs) excel in supervised image classification tasks BID17 ). Representation learned from such tasks can be transfer to other tasks, including object detection BID31 ; ; BID30 ) and semantic segmentation BID2 ; Badrinarayanan et al.) . Furthermore, if the training dataset is sufficiently larger, CNNs can improve the performance in classification, or learn better transferable representation, even if some labels are corrupted BID20 ; BID36 ; BID22 ).However , recent CNNs have far more parameters than their training samples. Therefore , the networks can memorize all the training data even if all labels are randomly replaced with the wrong ones ; ). This capability may degrade CNNs' performance under the label-corrupted situation, thus learning methods against label noise have been studied.Are CNNs robust or fragile to label noise? To investigate this question, we need to adopt noisy labels in controlled experiments. In previous work , both natural and synthetic noise have been used to research label corrupted situations. Natural noise appears in generally every dataset, and it comes from, for instance, annotators' mislabeling BID3 or their varieties BID6 ). Some researchers have been proposed robust training methods under this type of noise BID19 ; BID15 ; BID38 ). However, natural noise is uncontrollable, in other words, the relationship between the magnitude of noise and CNNs' performance has been unknown.On the other hand, synthetic noise simulates natural one by stochastically replacing ground truth labels with others. Class-independent uniform label permutation is a common setting BID15 ; BID10 ), yet some researchers use class-dependent label permutation, which is considered as more realistic situation BID26 ; BID8 ; BID27 ; BID9 ). Previous research has mainly adopted MNIST (10 classes, 60,000 training samples, BID18 ) or CIFAR-10/100 (10 and 100 classes, 50,000 training samples, BID17 ), and these datasets lack pre-defined conceptual relationships between classes. This limitation results in simplified noise simulation on such datasets, although synthetic noise enables researchers to research the relationship between the noise magnitude and the performance of networks.To investigate whether CNNs are robust or fragile to label corruption, we propose to use simulated noise considering possible mislabeling on ImageNet-1k (Russakovsky et al. (2015) ) to complement the disadvantages. Exploiting ImageNet-1k's conceptual hierarchy , we can divide its 1,000 labels into some clusters. We use these clusters to generate class-conditional label noise. We train several networks on the training dataset with and without corrupted labels. Then we evaluate the performance of the networks on the original validation set, the robustness of the networks against adversarial perturbation BID37 ; BID28 ), and their learned representation using transfer learning, canonical correlation analysis BID29 ; BID25 ).In this paper, we show the performance of CNNs trained on such synthesized noise considering possible mislabeling is better than uniformly synthesized noise, which is contrary to previous research BID8 ; BID27 ). Besides, models trained on class-dependent label noise are more robust to adversarial perturbation than ones trained on class-independent label noise. We also demonstrate CNNs trained under class-conditionally noisy conditions learn similar features to ones trained under the clean condition. As a result, even when 80% of labels are class-dependently corrupted, CNNs can learn useful representation for transfer learning. Meanwhile, we demonstrate class-independent noise leads models to learn different representation from ones trained with data with clean labels or label noise considering conceptual hierarchy. These differences can be attributed to the property of categorical cross entropy loss, which is a well-used loss function for image recognition tasks. We believe using class-independent noise is not a suitable protocol to investigate the CNNs' tolerance in practical situations. Why does class-dependent noise affect less than class-independent noise? We think there are two reasons: class-dependent noise is more informative, and it avoids the loss value getting too large.When class-dependent noise swaps a ground truth label with a wrong one, it is still a similar class to the original. Thus, the network can learn "which cluster the sample belongs to". This idea is related to the soft label BID12 ), though in our case, the label is "hard". Contrary to this, class-independent noise conveys no information.The other reason results from the property of categorical cross entropy loss. When the label of sample x is i, the loss value can be written as − log[f (x)] i , where f (x) is the corresponding softmax output. Therefore, when a CNN predicts x as i with weaker confidence, the penalty gets larger. Since the wrong label corrupted by class-dependent noise belongs to the same cluster as the ground truth, [f (x)] i is relatively large (c.f. Figure 2 (b) ). However, in the case of classindependent noise, the wrong label has nothing to do with the ground truth, and if the ground truth and the corrupted label are irrelevant, [f (x)] i should be small. Thus, the loss value gets larger, which leads the network to a worse solution.Also, our finding can be applicable to the quality control of annotation of data. Our results show class-dependent noise is more favorable than class-independent noise. Inexperienced but honest annotators will yield class-dependent noise, while lazy and malicious annotators may randomly annotate the labels and will yield class-independent noise. Therefore, according to our results, the administrators of the annotation need to exclude such workers. In this paper, we investigated the relationship between label noise, the performance and representation of CNNs in image classification tasks. We used ImageNet-1k with simulated noise which includes class-independent noise and class-dependent noise considering conceptual similarity. We examined such noise considering possible mislabeling causes less performance decrease and more robustness against adversarial perturbation compared to class-independent noise. Besides, we investigated the internal representation of CNNs trained with and without label corruption. Experiments showed networks trained on class-independently noisy data learn different representation from ones trained on clean or class-conditionally noisy data.Some previous research on label-noise-tolerant learning methods has used class-independent noise. However, as we revealed in this research, this noise setting is so artificial and straightforward that such methods may not be effective against real noise. Meanwhile, our results suggest plain CNNs themselves can be robust against real noise. This property should be good news for practitioners. Nevertheless, it is also shown noise considering possible mislabeling still somewhat degrades the performance of networks. Thus, how to avoid the effect of label noise is still a remaining problem.
Are CNNs robust or fragile to label noise? Practically, robust. The authors challenge the CNNs robustness to label noise using ImageNet 1k tree of WordNet. An analysis of convolutional neural network model performance when class dependent and class independent noise is introduced Demonstrates that CNNs are more robust to class-relevant label noise and argues that real-world noise should be class-relevant