A Generalization of Continuous Relaxation in Structured Pruning
Brad Larson Bishal Upadhyaya Luke McDermott Siddha Ganju
Deep learning harnesses massive parallel floating-point processing to train
and evaluate large neural networks. Trends indicate that deeper and larger
neural networks with an increasing number of parameters achieve higher accuracy
than smaller neural networks. This performance improvement, which often
requires heavy compute for both training and evaluation, eventually needs to
translate well to resource-constrained hardware for practical value. Structured
pruning asserts that while large networks enable us to find solutions to
complex computer vision problems, a smaller, computationally efficient
sub-network can be derived from the large neural network that retains model
accuracy but significantly improves computational efficiency.
  We generalize structured pruning with algorithms for network augmentation,
pruning, sub-network collapse and removal. In addition, we demonstrate
efficient and stable convergence up to 93% sparsity and 95% FLOPs reduction
without loss of inference accuracy using with continuous relaxation matching or
exceeding the state of the art for all structured pruning methods. The
resulting CNN executes efficiently on GPU hardware without computationally
expensive sparse matrix operations. We achieve this with routine automatable
operations on classification and segmentation problems using CIFAR-10,
ImageNet, and CityScapes datasets with the ResNet and U-NET network
architectures.