KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training
Truong Thao Nguyen Balazs Gerofi Edgar Josafat Martinez-Noriega François Trahay Mohamed Wahib
This paper proposes a method for hiding the least-important samples during
the training of deep neural networks to increase efficiency, i.e., to reduce
the cost of training. Using information about the loss and prediction
confidence during training, we adaptively find samples to exclude in a given
epoch based on their contribution to the overall learning process, without
significantly degrading accuracy. We explore the converge properties when
accounting for the reduction in the number of SGD updates. Empirical results on
various large-scale datasets and models used directly in image classification
and segmentation show that while the with-replacement importance sampling
algorithm performs poorly on large datasets, our method can reduce total
training time by up to 22% impacting accuracy only by 0.4% compared to the
baseline. Code available at https://github.com/TruongThaoNguyen/kakurenbo