Recently, there has been growing interest in methods that perform neural network compression, namely techniques that attempt to substantially reduce the size of a neural network without significant reduction in performance. However, most existing methods are post-processing approaches in that they take a learned neural network as input and output a compressed network by either forcing several parameters to take the same value (parameter tying via quantization) or pruning irrelevant edges (pruning) or both. In this paper, we propose a novel algorithm that jointly learns and compresses a neural network. The key idea in our approach is to change the optimization criteria by adding $k$ independent Gaussian priors over the parameters and a sparsity penalty. We show that our approach is easy to implement using existing neural network libraries, generalizes L1 and L2 regularization and elegantly enforces parameter tying as well as pruning constraints. Experimentally, we demonstrate that our new algorithm yields state-of-the-art compression on several standard benchmarks with minimal loss in accuracy while requiring little to no hyperparameter tuning as compared with related, competing approaches. Neural networks represent a family of highly flexible and scalable models that have rapidly achieved state-of-the-art performance in diverse domains including computer vision BID18 BID8 BID14 , speech BID5 , and sentiment analysis BID10 . Despite their successes, the storage requirements of large, modern neural networks make them impractical for certain applications with storage limitations (e.g., mobile devices). Moreover, as they are often trained on small datasets compared to their number of parameters (typically in the millions for state-of-the-art models), they can potentially overfit. In recent work, BID6 showed that a large proportion of neural network parameters are in fact not required for their generalization performance, and interest in model compression has surged.A variety of methods have been proposed to perform compression including pruning BID20 BID12 , quantization BID13 , lowrank approximation BID6 BID7 BID16 , group lasso BID27 , variational dropout BID23 , teacher-student training BID25 , etc. Here, we focus on the quantization/parameter tying approach to compression combined with pruning. Parameter tying assumptions occur naturally in the construction of convolutional neural networks (CNNs), but in these applications, the parameters to be tied are usually selected in advance of training. Recent work has focused on automatic parameter tying, i.e., automatically discovering which parameters of the model should be tied together. BID24 proposed a soft parameter tying scheme based on a mixtures of Gaussians prior and suggested a gradient descent method to jointly optimize both the weights in the network and the parameters of the mixture model. proposed a random parameter tying scheme based on hashing functions. BID13 proposed a compression pipeline that involved thresholding to prune low-magnitude parameters, k-means clustering to tie parameters layer-wise, and a final retraining stage to fine-tune tied parameters. This work demonstrated that high compression rates are achievable without much loss in accuracy. Building on the work of BID24 , K. Ullrich (2017) imposed a Gaussian mixture prior on the parameters to encourage clustering. At convergence, they proposed clustering the weights by assigning them to the mixture component that generates each weight with highest probability. BID22 proposed a full Bayesian approach to compression using scale mixture priors. This approach has the advantage that posterior distributions can be used to estimate the significance of individual bits in the learned weights. BID22 demonstrated that this approach can yield state-of-the-art compression results for some problems. BID1 recently proposed a soft-to-hard quantization approach in which scalar quantization is gradually learned through annealing a softened version of quantization distortion; compression is achieved with low-entropy parameter distribution instead of pruning.While much of the previous work has demonstrated that significant compression can be achieved while preserving the accuracy of the final network (in many cases ≈ 1% loss in accuracy), many of these approaches have potential drawbacks that can limit their applications. The Gaussian mixture approach of BID24 and K. Ullrich (2017) can be computationally expensive, as the time and memory requirements for backpropagation is increased K-fold under a K-component GMM prior, in addition to its large number of sensitive hyperparameters that can require extensive tuning. Moreover, the GMM objective itself suffers from well known local (and often pathological) minima issues. These local minimas are in addition to the ones encountered while training a neural network which in turn incurs high computational cost. The approach of BID13 uses separate pruning and parameter tying stages, which potentially limits its compression efficiency; additionally, the required layer-wise codebook storage can become expensive especially for deep networks. The parameter tying approach of is also only applied layerwise, and it typically requires more clusters, i.e., larger K, before the random weight sharing is effective (our experiments confirm that random parameter tying yields poor results when the number of distinct parameters is too small). The soft-to-hard quantization approach of BID1 resembles our method, but is essentially probabilistic like the GMM prior as it uses soft assignment for quantization which can be expensive. Finally, the full Bayesian approach, similar to the GMM approach, has a number of additional parameters to tune (e.g., constraints on variances, careful initialization of each of the variational parameters, etc.). The Bayesian approach also requires sampling for prediction (which can be done deterministically but with some additional loss). We hope to argue that such sophisticated methods may not be necessary to achieve good compression in practice.The approach to compression in this work uses quantization and sparsity inducing priors. For quantization, we consider an independent Gaussian prior, that is, each parameter is non-probabilistically assigned to one of K independent Gaussian distributions, and the prior penalizes each weight by its 2 distance to the mean of its respective Gaussian. This prior places no restriction on which weights can be tied together (e.g., weights from the input could be tied to weights into the output), reduces the number of hyperparameters that need to be tuned compared to probabilistic methods like Gaussian mixtures, and requires only a small change to the typical gradient descent updates with only linear time and memory overhead. We observe that quantization alone is not enough to achieve the desired level of compression, and introduce pruning by adding a standard 1 penalty on top of the quantization prior; we demonstrate experimentally that the combined prior yields state-of-the-art compression results on standard benchmark data sets.
A k-means prior combined with L1 regularization yields state-of-the-art compression results. This paper explores soft parameter tying and compression of DNNs/CNNs