Pruning units in a deep network can help speed up inference and training as well as reduce the size of the model. We show that bias propagation is a pruning technique which consistently outperforms the common approach of merely removing units,  regardless of the architecture and the dataset.   We also show how a simple adaptation to an existing scoring function allows us to select the best units to prune.   Finally,  we show that the units selected by the best performing scoring functions are somewhat consistent over the course of training, implying the dead parts of the network appear during the stages of training. Pruning is a successful method for reducing the size of a trained neural network and accelerating inference. Pruning consists of deleting the parts of the network whose removal least affects the network performance. Many pruning methods proposed in the literature differ in computational cost and in effectiveness in ways that are hard to assess.In an interesting recent work, BID3 argue for the so called "winning ticket" hypothesis. More precisely, they train a large network after saving the random initial value of each parameter. After training, they prune the large network to produce a smaller network with one fifth of the weights. Setting its weights to their saved initial values and retraining achieves a performance close to that of the large trained network with a much reduced computational cost. This result opens up a new frontier for pruning methods, where they are used to detect useless units early in the training and therefore accelerating the inference.This contribution studies the effect of pruning methods throughout the training process. We also present mean replacement, a unit pruning method that extends the idea of bias propagation introduced in (Ye et al., 2018) to the non-constrained training setting. The main observations of our work can then be summarized as follows:• Regardless of the scoring function used, bias propagation reduces the pruning penalty for networks without batch normalization.• Fine-tuning the pruned network with additional training iterations reduces the bias propagation advantage but not very quickly.• Absolute valued approximation of the pruning penalty provides superior performance over the normal first order approximation. This finding confirms the observations made by BID11 .• Units that are selected by the best performing scoring function seem to come from a small subset of units. This finding confirms BID3 's comments on the lottery ticket and BID2 's claims about dead units.The rest of the paper is organized as follows. After reviewing the related work in Section 2. we define our pruning methods and scoring functions in Section 3. Section 4 provides an empirical evaluation comparing various combinations of scoring functions and methods under varying pruning fractions, datasets, and models. We briefly provide some concluding remarks and discuss future work in Section 5. This work presents an experimental comparison of unit pruning strategies throughout the training process. We introduce the mean replacement approach and show that it substantially reduces the impact of the unit removal on the loss function. We also show that fine-tuning the pruned networks does not reduce the mean replacement advantage very quickly. We argue that direct first order approximation of the pruning penalty are poor predictors of the pruning penalty incurred by the simultaneous removal of multiple units because the neglected high order terms can become significant. In contrast the absolute value versions of these approximations achieve the best performance. Finally we provide some evidence showing that our best pruning methods identify a stable set of prunable units relatively early in the training process.This last observation begs for future work. Can we combine pruning and training in a manner that reduces the computational training cost to a quantity comparable to training the "winning ticket" network? If we decided that we will be using Mean Replacement as our pruning method, we can define a new scoring function, i.e. the first order Taylor approximation of the pruning penalty after mean replacement. We name this new saliency function as Mean Replacement Saliency (MRS) Let us parameterize the loss as a function with activations and write down the first order approximation of the absolute change in the loss. DISPLAYFORM0 where DISPLAYFORM1 If we were interested in the average change in the loss we can write down the Equation 5 without the absolute values. In other words approximations on absolute change penalizes both directions, emphasizing the change in the neural network itself rather then the loss function.
Mean Replacement is an efficient method to improve the loss after pruning and Taylor approximation based scoring functions works better with absolute values.  Proposes a simple improvement to methods for unit pruning using "mean replacement" This paper presents a mean-replacement pruning strategy and utilizes the absolute-valued Taylor expansion as the scoring function for the pruning