Group channel pruning and spatial attention distilling for object
  detection
Yun Chu Pu Li Yong Bai Zhuhua Hu Yongqing Chen Jiafeng Lu
Due to the over-parameterization of neural networks, many model compression
methods based on pruning and quantization have emerged. They are remarkable in
reducing the size, parameter number, and computational complexity of the model.
However, most of the models compressed by such methods need the support of
special hardware and software, which increases the deployment cost. Moreover,
these methods are mainly used in classification tasks, and rarely directly used
in detection tasks. To address these issues, for the object detection network
we introduce a three-stage model compression method: dynamic sparse training,
group channel pruning, and spatial attention distilling. Firstly, to select out
the unimportant channels in the network and maintain a good balance between
sparsity and accuracy, we put forward a dynamic sparse training method, which
introduces a variable sparse rate, and the sparse rate will change with the
training process of the network. Secondly, to reduce the effect of pruning on
network accuracy, we propose a novel pruning method called group channel
pruning. In particular, we divide the network into multiple groups according to
the scales of the feature layer and the similarity of module structure in the
network, and then we use different pruning thresholds to prune the channels in
each group. Finally, to recover the accuracy of the pruned network, we use an
improved knowledge distillation method for the pruned network. Especially, we
extract spatial attention information from the feature maps of specific scales
in each group as knowledge for distillation. In the experiments, we use YOLOv4
as the object detection network and PASCAL VOC as the training dataset. Our
method reduces the parameters of the model by 64.7 % and the calculation by
34.9%.