Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approaches. This work proposes a progressive weight pruning approach based on ADMM (Alternating Direction Method of Multipliers), a powerful technique to deal with non-convex optimization problems with potentially combinatorial constraints. Motivated by dynamic programming, the proposed method reaches extremely high pruning rate by using partial prunings with moderate pruning rates. Therefore, it resolves the accuracy degradation and long convergence time problems when pursuing extremely high pruning ratios. It achieves up to 34× pruning rate for ImageNet dataset and 167× pruning rate for MNIST dataset, significantly higher than those reached by the literature work. Under the same number of epochs, the proposed method also achieves faster convergence and higher compression rates. The codes and pruned DNN models are released in the anonymous link bit.ly/2zxdlss. Deep neural networks (DNNs) have achieved human-level performance in many application domains such as image classification BID17 , object recognition BID18 BID9 , natural language processing , etc. At the same time, the networks are growing deeper and bigger for higher classification/recognition performance (i.e., accuracy) BID25 . However, the very large DNN model size increases the computation time of the inference phase. To make matters worse, the large model size hinders DNN' deployments on edge computing, which provides the ubiquitous application scenarios of DNNs besides cloud computing applications.As a result, extensive research efforts have been devoted to DNN model compression, in which DNN weight pruning is a representative technique. BID7 is the first work to present the DNN weight pruning method, which prunes the weights with small magnitudes and retrains the network model, heuristically and iteratively. After that, more sophisticated heuristics have been proposed for DNN weight pruning, e.g., incorporating both weight pruning and growing BID6 , L 1 regularization BID27 , and genetic algorithms BID3 . Other improvement directions of weight pruning include trading-off between accuracy and compression rate, e.g., energy-aware pruning BID29 , incorporating regularity, e.g., channel pruning BID10 , and structured sparsity learning BID27 .While the weight pruning technique explores the redundancy in the number of weights of a network model, there are other sources of redundancy in a DNN model. For example , the weight quantization BID19 BID22 BID34 BID20 BID23 BID13 BID1 and clustering BID8 techniques explore the redundancy in the number of bits for weight representation. The activation pruning technique BID15 BID24 leverages the redundancy in the intermediate results. While our work focuses on weight pruning as the major DNN model compression technique, it is orthogonal to the other model compression techniques and might be integrated under a single ADMM-based framework for achieving more compact network models.The majority of prior work on DNN weight pruning take heuristic approaches to reduce the number of weights as much as possible, while preserving the expressive power of the DNN model. Then one may ask , how can we push for the utmost sparsity of the DNN model without hurting accuracy? and what is the maximum compression rate we can achieve by weight pruning? Towards this end , BID32 took a tentative step by proposing an optimization-based approach that leverages ADMM (Alternating Direction Method of Multipliers), a powerful technique to deal with nonconvex optimization problems with potentially combinatorial constraints. This direct ADMM-based weight pruning technique can be perceived as a smart DNN regularization where the regularization target is dynamically changed in each ADMM iteration. As a result it achieves higher compression (pruning) rate than heuristic methods.Inspired by BID32 , in this paper we propose a progressive weight pruning approach that incorporates both an ADMM-based algorithm and masked retraining, and takes a progressive means targeting at extremely high compression (pruning) rates with negligible accuracy loss. The contributions of this work are summarized as follows:• We make a key observation that when pursuing extremely high compression rates (say 150× for LeNet-5 or 30× for AlexNet), the direct ADMM-based weight pruning approach BID32 cannot produce exactly sparse models upon convergence, in that many weights to be pruned are close to zero but not exactly zero. Certain accuracy degradation will result from this phenomenon if we simply set these weights to zero.• We propose and implement the progressive weight pruning paradigm that reaches an extremely high compression rate through multiple partial prunings with progressive pruning rates. This progressive approach, motivated by dynamic programming, helps to mitigate the long convergence time by direct ADMM pruning.• Extensive experiments are performed by comparing with many state-of-the-art weight pruning approaches and the highest compression rates in the literature are achieved by our progressive weight pruning framework, while the loss of accuracy is kept negligible. Our method achieves up to 34× pruning rate for the ImageNet data set and 167× pruning rate for the MNIST data set, with virtually no accuracy loss. Under the same number of epochs, the proposed method achieves notably better convergence and higher compression rates than prior iterative pruning and direct ADMM pruning methods.We provide codes (both Caffe and TensorFlow versions) and pruned DNN models (both for the ImageNet and MNIST data sets) in the link: bit.ly/2zxdlss. For other types of DNN models, we have tested the proposed method on the facial recognition application on two representative DNN models BID16 BID12 . We demonstrate over 10× weight pruning rate with 0.2% and 0.4% accuracy loss, respectively, compared with the original DNN models.In summary, the experimental results demonstrate that our framework applies to a broad set of representative DNN models and consistently outperforms the prior work. It also applies to the DNN models that consist of mainly convolutional layers, which are different with weight pruning using prior methods. These promising results will significantly contribute to the energy-efficient implementation of DNNs in mobile and embedded systems, and on various hardware platforms.Finally, some recent work have focused on the simultaneous weight pruning and weight quantization, as both will contribute to the model storage compression of DNNs. Weight pruning and quantization can be unified under the ADMM framework, and we demonstrate the comparison results in TAB8 using the LeNet-5 model as illustrative example. As can be observed in the table, we can simultaneously achieve 167× weight reduction and use 2-bit for fully-connected layer weight quantization and 3-bit for convolutional layer weight quantization. The overall accuracy is 99.0%. When we focus on the weight data storage, the compression rate is unprecendented 1,910× compared with the original DNN model with floating point representation. When indices (required in weight pruning) are accounted for, the overall compression rate is 623×, which is still much higher than the prior work. It is interesting to observe that the amount of storage for indices is even higher than that for actual weight data. This work proposes a progressive weight pruning approach based on ADMM, a powerful technique to deal with non-convex optimization problems with potentially combinatorial constraints. Motivated by dynamic programming, the proposed method reaches extremely high pruning rates by using partial prunings, with moderate pruning rates in each partial pruning step. Therefore, it resolves the accuracy degradation and long convergence time problems when pursuing extremely high pruning ratios. It achieves up to 34× pruning rate for the ImageNet data set and 167× pruning rate for the MNIST data set, significantly higher than those reached by work in the existing literature. Under the same number of epochs, the proposed method also achieves better convergence and higher compression rates.
We implement a DNN weight pruning approach that achieves the highest pruning rates. This paper focuses on weight pruning for neural network compression, achiving 30x compression rate for AlexNet and VGG for ImageNet. A progressive pruning technique which imposes structural sparsity constraint on the weight parameter and rewrites the optimization as an ADMM framework, achieving higher accurancy than projected gradient descent.