Human labeling errors and their impact on ConvNets for satellite image
  scene classification
Longkang Peng Tao Wei Xuehong Chen Xiaobei Chen Rui Sun Luoma Wan Xiaolin Zhu
Convolutional neural networks (ConvNets) have been successfully applied to
satellite image scene classification. Human-labeled training datasets are
essential for ConvNets to perform accurate classification. Errors in
human-labeled training datasets are unavoidable due to the complexity of
satellite images. However, the distribution of human labeling errors on
satellite images and their impact on ConvNets have not been investigated. To
fill this research gap, this study, for the first time, collected real-world
labels from 32 participants and explored how their errors affect three ConvNets
(VGG16, GoogleNet and ResNet-50) for high-resolution satellite image scene
classification. We found that: (1) human labeling errors have significant class
and instance dependence, which is fundamentally different from the simulation
noise in previous studies; (2) regarding the overall accuracy of all classes,
when human labeling errors in training data increase by one unit, the overall
accuracy of ConvNets classification decreases by approximately half a unit; (3)
regarding the accuracy of each class, the impact of human labeling errors on
ConvNets shows large heterogeneity across classes. To uncover the mechanism
underlying the impact of human labeling errors on ConvNets, we further compared
it with two types of simulated labeling noise: uniform noise (errors
independent of both classes and instances) and class-dependent noise (errors
independent of instances but not classes). Our results show that the impact of
human labeling errors on ConvNets is similar to that of the simulated
class-dependent noise but not to that of the simulated uniform noise,
suggesting that the impact of human labeling errors on ConvNets is mainly due
to class-dependent errors rather than instance-dependent errors.