Auto-Train-Once: Controller Network Guided Automatic Network Pruning
  from Scratch
Xidong Wu Shangqian Gao Zeyu Zhang Zhenzhen Li Runxue Bao Yanfu Zhang Xiaoqian Wang Heng Huang
Current techniques for deep neural network (DNN) pruning often involve
intricate multi-step processes that require domain-specific expertise, making
their widespread adoption challenging. To address the limitation, the
Only-Train-Once (OTO) and OTOv2 are proposed to eliminate the need for
additional fine-tuning steps by directly training and compressing a general DNN
from scratch. Nevertheless, the static design of optimizers (in OTO) can lead
to convergence issues of local optima. In this paper, we proposed the
Auto-Train-Once (ATO), an innovative network pruning algorithm designed to
automatically reduce the computational and storage costs of DNNs. During the
model training phase, our approach not only trains the target model but also
leverages a controller network as an architecture generator to guide the
learning of target model weights. Furthermore, we developed a novel stochastic
gradient algorithm that enhances the coordination between model training and
controller network training, thereby improving pruning performance. We provide
a comprehensive convergence analysis as well as extensive experiments, and the
results show that our approach achieves state-of-the-art performance across
various model architectures (including ResNet18, ResNet34, ResNet50, ResNet56,
and MobileNetv2) on standard benchmark datasets (CIFAR-10, CIFAR-100, and
ImageNet).