Deep convolutional neural networks (CNNs) are deployed in various applications but demand immense computational requirements. Pruning techniques and Winograd convolution are two typical methods to reduce the CNN computation. However, they cannot be directly combined because Winograd transformation fills in the sparsity resulting from pruning. Li et al. (2017) propose sparse Winograd convolution in which weights are directly pruned in the Winograd domain, but this technique is not very practical because Winograd-domain retraining requires low learning rates and hence significantly longer training time. Besides, Liu et al. (2018) move the ReLU function into the Winograd domain, which can help increase the weight sparsity but requires changes in the network structure. To achieve a high Winograd-domain weight sparsity without changing network structures, we propose a new pruning method, spatial-Winograd pruning. As the first step, spatial-domain weights are pruned in a structured way, which efficiently transfers the spatial-domain sparsity into the Winograd domain and avoids Winograd-domain retraining. For the next step, we also perform pruning and retraining directly in the Winograd domain but propose to use an importance factor matrix to adjust weight importance and weight gradients. This adjustment makes it possible to effectively retrain the pruned Winograd-domain network without changing the network structure. For the three models on the datasets of CIFAR-10, CIFAR-100, and ImageNet, our proposed method can achieve the Winograd-domain sparsities of 63%, 50%, and 74%, respectively. Deep convolutional neural networks (CNNs) have been ubiquitously utilized in various application domains. However, their performance comes at the cost of a significant amount of computation which keeps growing over time. As an example, for the ImageNet challenge BID12 , BID5 proposed AlexNet which requires more than 1.1 × 10 9 multiplications. Later, in 2016, the ResNet-152 model BID3 increased the computation cost to 11.3 × 10 9 multiplications. This high computation cost limits the deployment of larger and deeper CNN models.There are two primary methods to reduce the required computation of CNN models: pruning techniques and Winograd/FFT convolution. Pruning removes redundant weight parameters, inducing sparsity into the network. On the other hand, Winograd convolution BID6 and FFT convolution BID10 transform the computation into different domains. The convolution operations can then be replaced by element-wise multiplications. For the typical convolution kernel size of 3 × 3, Winograd convolution can achieve more than twofold speedup over highly optimized spatial convolution algorithms, and typically requires fewer flops than FFT-based approaches BID7 . Therefore, in this paper, we focus on the Winograd convolution.The pruning techniques and Winograd convolution are not directly compatible with each other. Sparse weight matrices, which are generated by pruning, lose most of the sparsity after the Winograd transformation from the spatial (original) domain to the Winograd domain. The remaining sparsity is much lower than what we need for improving computation performance.To increase the Winograd-domain sparsity, BID7 propose to perform pruning and retraining directly on Winograd-domain weights. However, it requires using an extremely small learning rate, e.g., 200x smaller for AlexNet, in retraining and is difficult to be applied to deep networks. Besides, Winograd-ReLU pruning BID9 moves ReLU function into the Winograd domain, which helps increase Winograd-domain sparsity but requires changes in the network structure.In this paper, to further improve the sparsity of Winograd-domain weights without changing the network structure, we propose a new pruning method, spatial-Winograd pruning. It includes two parts: spatial structured pruning and Winograd direct pruning. In spatial structured pruning, we prune the spatial-domain weights in a structured way, in which the structures are designed to transfer the spatial-domain sparsity into the Winograd domain efficiently. After spatial structured pruning, weights of the pruned layers will be converted to and kept in the Winograd domain. Then, for Winograd direct pruning, we perform pruning and retraining entirely in the Winograd domain to improve the sparsity further. This paper makes the following contributions:• We propose a new pruning method, spatial-Winograd pruning. Without changing the network structure, it can achieve higher sparsity in Winograd-domain weights compared with previous methods.• As the first part of spatial-Winograd pruning, we provide a structured pruning method to transfer the spatial-domain sparsity into the Winograd domain efficiently. It can help avoid Winograd-domain retraining in this part and accelerate the pruning process.• In the second part, to perform pruning directly in the Winograd domain, we present a new approach to measuring the importance of each Winograd-domain weight based on its impact on output activations. Also , we propose to use an importance factor matrix to adjust the gradients of Winograd-domain weights, which makes it much faster to retrain deep networks directly in the Winograd domain without changing the network structure. In this paper, we present a new pruning method, spatial-Winograd pruning, to improve the Winograd-domain weight sparsity without changing network structures. It includes two steps: spatial structured pruning and Winograd direct pruning. In spatial structured pruning, we prune the spatial-domain weights based on the internal structure in the Winograd transformation. It can help efficiently transfer the spatial-domain sparsity into the Winograd domain. For Winograd direct pruning, we perform both pruning and retraining in the Winograd domain. An importance factor matrix is proposed to adjust the weight gradients in Winograd retraining, which makes it possible to effectively retrain the Winograd-domain network to regain the original accuracy without changing the network structure. We evaluate spatial-Winograd pruning on three datasets, CIFAR-10, CIFAR-100, ImageNet, and it can achieve the Winograd-domain sparsities of 63%, 50%, and 74%, respectively.
To accelerate the computation of convolutional neural networks, we propose a new two-step pruning technique which achieves a higher Winograd-domain weight sparsity without changing the network structure. Proposes a spatial-Winograd pruning framework which allows pruned weight from the spatial domain to be kept in the Winograd domain and improves the sparsity of the Winograd domain. Proposes two techniques for pruning convolutional layers which use the Winograd algorithm