Towards Higher Ranks via Adversarial Weight Pruning
Yuchuan Tian Hanting Chen Tianyu Guo Chao Xu Yunhe Wang
Convolutional Neural Networks (CNNs) are hard to deploy on edge devices due
to its high computation and storage complexities. As a common practice for
model compression, network pruning consists of two major categories:
unstructured and structured pruning, where unstructured pruning constantly
performs better. However, unstructured pruning presents a structured pattern at
high pruning rates, which limits its performance. To this end, we propose a
Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an
adversarial manner. In each step, we minimize the low-rank approximation error
for the weight matrices using singular value decomposition, and maximize their
distance by pushing the weight matrices away from its low rank approximation.
This rank-based optimization objective guides sparse weights towards a
high-rank topology. The proposed method is conducted in a gradual pruning
fashion to stabilize the change of rank during training. Experimental results
on various datasets and different tasks demonstrate the effectiveness of our
algorithm in high sparsity. The proposed RPG outperforms the state-of-the-art
performance by 1.13% top-1 accuracy on ImageNet in ResNet-50 with 98% sparsity.
The codes are available at
https://github.com/huawei-noah/Efficient-Computing/tree/master/Pruning/RPG and
https://gitee.com/mindspore/models/tree/master/research/cv/RPG.