Revisiting Data Augmentation in Model Compression: An Empirical and
  Comprehensive Study
Muzhou Yu Linfeng Zhang Kaisheng Ma
The excellent performance of deep neural networks is usually accompanied by a
large number of parameters and computations, which have limited their usage on
the resource-limited edge devices. To address this issue, abundant methods such
as pruning, quantization and knowledge distillation have been proposed to
compress neural networks and achieved significant breakthroughs. However, most
of these compression methods focus on the architecture or the training method
of neural networks but ignore the influence from data augmentation. In this
paper, we revisit the usage of data augmentation in model compression and give
a comprehensive study on the relation between model sizes and their optimal
data augmentation policy. To sum up, we mainly have the following three
observations: (A) Models in different sizes prefer data augmentation with
different magnitudes. Hence, in iterative pruning, data augmentation with
varying magnitudes leads to better performance than data augmentation with a
consistent magnitude. (B) Data augmentation with a high magnitude may
significantly improve the performance of large models but harm the performance
of small models. Fortunately, small models can still benefit from strong data
augmentations by firstly learning them with "additional parameters" and then
discard these "additional parameters" during inference. (C) The prediction of a
pre-trained large model can be utilized to measure the difficulty of data
augmentation. Thus it can be utilized as a criterion to design better data
augmentation policies. We hope this paper may promote more research on the
usage of data augmentation in model compression.