Importance Estimation with Random Gradient for Neural Network Pruning
Suman Sapkota Binod Bhattarai
Global Neuron Importance Estimation is used to prune neural networks for
efficiency reasons. To determine the global importance of each neuron or
convolutional kernel, most of the existing methods either use activation or
gradient information or both, which demands abundant labelled examples. In this
work, we use heuristics to derive importance estimation similar to Taylor First
Order (TaylorFO) approximation based methods. We name our methods TaylorFO-abs
and TaylorFO-sq. We propose two additional methods to improve these importance
estimation methods. Firstly, we propagate random gradients from the last layer
of a network, thus avoiding the need for labelled examples. Secondly, we
normalize the gradient magnitude of the last layer output before propagating,
which allows all examples to contribute similarly to the importance score. Our
methods with additional techniques perform better than previous methods when
tested on ResNet and VGG architectures on CIFAR-100 and STL-10 datasets.
Furthermore, our method also complements the existing methods and improves
their performances when combined with them.