We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ regularized objective is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by ``stretching'' a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer. Deep neural networks are flexible function approximators that have been very successful in a broad range of tasks. They can easily scale to millions of parameters while allowing for tractable optimization with mini-batch stochastic gradient descent (SGD), graphical processing units (GPUs) and parallel computation. Nevertheless they do have drawbacks. Firstly, it has been shown in recent works BID6 that they are greatly overparametrized as they can be pruned significantly without any loss in accuracy; this exhibits unnecessary computation and resources. Secondly, they can easily overfit and even memorize random patterns in the data BID39 , if not properly regularized. This overfitting can lead to poor generalization in practice.A way to address both of these issues is by employing model compression and sparsification techniques. By sparsifying the model, we can avoid unnecessary computation and resources, since irrelevant degrees of freedom are pruned away and do not need to be computed. Furthermore, we reduce its complexity, thus penalizing memorization and alleviating overfitting.A conceptually attractive approach is the L 0 norm regularization of (blocks of) parameters; this explicitly penalizes parameters for being different than zero with no further restrictions. However, the combinatorial nature of this problem makes for an intractable optimization for large models.In this paper we propose a general framework for surrogate L 0 regularized objectives. It is realized by smoothing the expected L 0 regularized objective with continuous distributions in a way that can maintain the exact zeros in the parameters while still allowing for efficient gradient based optimization. This is achieved by transforming continuous random variables (r.v.s) with a hard nonlinearity, the Figure 1 : L p norm penalties for a parameter θ according to different values of p. It is easily observed that both weight decay and Lasso, p = 2 and p = 1 respectively, impose shrinkage for large values of θ. By gradually allowing p < 1 we observe that the shrinkage is reduced and at the limit of p = 0 we observe that the penalty is a constant for θ = 0.hard-sigmoid. We further propose and employ a novel distribution obtained by this procedure; the hard concrete. It is obtained by "stretching" a binary concrete random variable BID19 BID12 and then passing its samples through a hard-sigmoid. We demonstrate the effectiveness of this simple procedure in various experiments.2 MINIMIZING THE L 0 NORM OF PARAMETRIC MODELS One way to sparsify parametric models, such as deep neural networks, with the least assumptions about the parameters is the following; let D be a dataset consisting of N i.i.d. input output pairs {(x 1 , y 1 ), . . . , (x N , y N )} and consider a regularized empirical risk minimization procedure with an L 0 regularization on the parameters θ of a hypothesis (e.g. a neural network) h(·; θ) 1 : DISPLAYFORM0 DISPLAYFORM1 where |θ| is the dimensionality of the parameters, λ is a weighting factor for the regularization and L(·) corresponds to a loss function, e.g. cross-entropy loss for classification or mean-squared error for regression. The L 0 norm penalizes the number of non-zero entries of the parameter vector and thus encourages sparsity in the final estimates θ * . The Akaike Information Criterion (AIC) BID0 ) and the Bayesian Information Criterion (BIC) BID28 , well-known model selection criteria, correspond to specific choices of λ. Notice that the L 0 norm induces no shrinkage on the actual values of the parameters θ; this is in contrast to e.g. L 1 regularization and the Lasso BID32 , where the sparsity is due to shrinking the actual values of θ. We provide a visualization of this effect in Figure 1 .Unfortunately , optimization under this penalty is computationally intractable due to the nondifferentiability and combinatorial nature of 2 |θ| possible states of the parameter vector θ. How can we relax the discrete nature of the L 0 penalty such that we allow for efficient continuous optimization of Eq. 1, while allowing for exact zeros in the parameters? This section will present the necessary details of our approach. We have described a general recipe that allows for optimizing the L 0 norm of parametric models in a principled and effective manner. The method is based on smoothing the combinatorial problem with continuous distributions followed by a hard-sigmoid. To this end, we also proposed a novel distribution which we coin as the hard concrete; it is a "stretched" binary concrete distribution, the samples of which are transformed by a hard-sigmoid. This in turn better mimics the binary nature of Bernoulli distributions while still allowing for efficient gradient based optimization. In experiments we have shown that the proposed L 0 minimization process leads to neural network sparsification that is competitive with current approaches while theoretically allowing for speedup in training. We have further shown that this process can provide a good inductive bias and regularizer, as on the CIFAR experiments with wide residual networks we improved upon dropout.As for future work; better harnessing the power of conditional computation for efficiently training very large neural networks with learned sparsity patterns is a potential research direction. It would be also interesting to adopt a full Bayesian treatment over the parameters θ, such as the one employed at ; BID18 . This would then allow for further speedup and compression due to the ability of automatically learning the bit precision of each weight. Finally, it would be interesting to explore the behavior of hard concrete r.v.s at binary latent variable models, since they can be used as a drop in replacement that allow us to maintain both the discrete nature as well as the efficient reparametrization gradient optimization.
We show how to optimize the expected L_0 norm of parametric models with gradient descent and introduce a new distribution that facilitates hard gating. The authors introduce a gradient-based approach to minimize an objective function with an L0 sparse penalty to help learn sparse neural networks