With the rapidly scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for efficient deployment. This work aims to advance the compression beyond the weights to the activations of DNNs. We propose the Integral Pruning (IP) technique which integrates the activation pruning with the weight pruning. Through the learning on the different importance of neuron responses and connections, the generated network, namely IPnet, balances the sparsity between activations and weights and therefore further improves execution efficiency. The feasibility and effectiveness of IPnet are thoroughly evaluated through various network models with different activation functions and on different datasets. With <0.5% disturbance on the testing accuracy, IPnet saves 71.1% ~ 96.35% of computation cost, compared to the original dense models with up to 5.8x and 10x reductions in activation and weight numbers, respectively. Deep neural networks (DNNs) have demonstrated significant advantages in many real-world applications, such as image classification, object detection and speech recognition BID6 BID15 BID16 . On the one hand, DNNs are developed for improving performance in these applications, which leads to intensive demands in data storage, communication and processing. On the other hand, the ubiquitous intelligence promotes the deployment of DNNs in light-weight embedded systems that are equipped with only limited memory and computation resource. To reduce the model size while ensuring the performance quality, DNN pruning is widely explored. Redundant weight parameters are removed by zeroing-out those in small values BID4 BID13 . Utilizing the zero-skipping technique BID5 on sparse weight parameters can further save the computation cost. In addition, many specific DNN accelerator designs BID0 BID14 leveraged the intrinsic zero-activation pattern of the rectified linear unit (ReLU) to realize the activation sparsity. The approach, however, cannot be directly extended to other activation functions, e.g., leaky ReLU.Although these techniques achieved tremendous success, pruning only the weights or activations cannot lead to the best inference speed, which is a crucial metric in DNN deployment, for the following reasons. First, the existing weight pruning methods mainly focus on the model size reduction. However, the most essential challenge of speeding up DNNs is to minimize the computation cost, such as the intensive multiple-and-accumulate operations (MACs). Particularly, the convolution (conv) layers account for most of the computation cost and dominate the inference time in DNNs BID13 . Because weights are shared in convolution, the execution speed of conv layers is usually bounded by computation instead of memory accesses BID7 BID21 . Second, the activation in DNNs is not strictly limited with ReLU. The intrinsic zeroactivation patterns do not exist in non-ReLU activation functions, such as leaky ReLU and sigmoid. Third, the weights and activations of a network together determine the network performance. Our experiment shows that the zero-activation percentage obtained by ReLU decreases after applying the weight pruning BID5 . Such a deterioration in activation sparsity could potentially eliminate the advantage of the aforementioned accelerator designs.In this work, we propose the integral pruning (IP) technique to minimize the computation cost of DNNs by pruning both weights and activations. As the pruning processes for weights and activations are correlated, IP learns dynamic activation masks by attaching activation pruning to weight pruning after static weight masks are well trained. Through the learning on the different importance of neuron responses and connections, the generated network, namely IPnet, balances the sparsity between activations and weights and therefore further improves execution efficiency. Moreover, our method not only stretches the intrinsic activation sparsity of ReLU, but also targets as a general approach for other activation functions, such as leaky ReLU. Our experiments on various network models with different activation functions and on different datasets show substantial reduction in MACs by the proposed IPnet. Compared to the original dense models, IPnet can obtain up to 5.8× activation compression rate, 10× weight compression rate and eliminate 71.1% ∼ 96.35% of MACs. Compared to state-of-the-art weight pruning technique BID4 , IPnet can further reduce the computation cost 1.2× ∼ 2.7×. The static activation pruning approach has been widely adopted in efficient DNN accelerator designs BID0 BID14 . By selecting a proper static threshold θ in Equation (2), more activations can be pruned with little impact on model accuracy. For the activation pruning in IP, the threshold is dynamically set according to the winner rate and activation distribution layer-wise. The comparison between static and dynamic pruning is conducted on ResNet-32 for CIFAR-10 dataset. For the static pruning setup, the θ for leaky ReLU is assigned in the range of [0.07, 0.14], which brings different activation sparsity patterns. To minimize the computation cost in DNNs, IP combining weight pruning and activation pruning is proposed in this paper. The experiment results on various models for MNIST, CIFAR-10 and ImageNet datasets have demonstrated considerable computation cost reduction. In total, a 2.3× -5.8× activation compression rate and a 2.5× -10× weight compression rate are obtained. Only 3.65% -28.9% of MACs are left with marginal effects on model accuracy, which outperforms the weight pruning by 1.2× -2.7×. The IPnets are targeted for the dedicated DNN accelerator designs with efficient sparse matrix storage and computation units on chip. The IPnets featuring compressed model size and reduced computation cost will meet the constraints from memory space and computing resource in embedded systems.
This work advances DNN compression beyond the weights to the activations by integrating the activation pruning with the weight pruning.  An integral model compression method that handles both weight and activation pruning, leading to more efficient network computation and effective reduction of the number of multiply-and-accumulate. This article presents a novel approach to reduce the computation cost of deep neural networks by integrating activation pruning along with weight pruning and show that common techniques of exclusive weight pruning  increases the number of non-zero activations after ReLU.