On the Efficacy of Multi-scale Data Samplers for Vision Applications
Elvis Nunez Thomas Merth Anish Prabhu Mehrdad Farajtabar Mohammad Rastegari Sachin Mehta Maxwell Horton
Multi-scale resolution training has seen an increased adoption across
multiple vision tasks, including classification and detection. Training with
smaller resolutions enables faster training at the expense of a drop in
accuracy. Conversely, training with larger resolutions has been shown to
improve performance, but memory constraints often make this infeasible. In this
paper, we empirically study the properties of multi-scale training procedures.
We focus on variable batch size multi-scale data samplers that randomly sample
an input resolution at each training iteration and dynamically adjust their
batch size according to the resolution. Such samplers have been shown to
improve model accuracy beyond standard training with a fixed batch size and
resolution, though it is not clear why this is the case. We explore the
properties of these data samplers by performing extensive experiments on
ResNet-101 and validate our conclusions across multiple architectures, tasks,
and datasets. We show that multi-scale samplers behave as implicit data
regularizers and accelerate training speed. Compared to models trained with
single-scale samplers, we show that models trained with multi-scale samplers
retain or improve accuracy, while being better-calibrated and more robust to
scaling and data distribution shifts. We additionally extend a multi-scale
variable batch sampler with a simple curriculum that progressively grows
resolutions throughout training, allowing for a compute reduction of more than
30%. We show that the benefits of multi-scale training extend to detection and
instance segmentation tasks, where we observe a 37% reduction in training FLOPs
along with a 3-4% mAP increase on MS-COCO using a Mask R-CNN model.