Convolutional Neural Network Compression via Dynamic Parameter Rank
  Pruning
Manish Sharma Jamison Heard Eli Saber Panos P. Markopoulos
While Convolutional Neural Networks (CNNs) excel at learning complex
latent-space representations, their over-parameterization can lead to
overfitting and reduced performance, particularly with limited data. This,
alongside their high computational and memory demands, limits the applicability
of CNNs for edge deployment. Low-rank matrix approximation has emerged as a
promising approach to reduce CNN parameters, but its application presents
challenges including rank selection and performance loss. To address these
issues, we propose an efficient training method for CNN compression via dynamic
parameter rank pruning. Our approach integrates efficient matrix factorization
and novel regularization techniques, forming a robust framework for dynamic
rank reduction and model compression. We use Singular Value Decomposition (SVD)
to model low-rank convolutional filters and dense weight matrices and we
achieve model compression by training the SVD factors with back-propagation in
an end-to-end way. We evaluate our method on an array of modern CNNs, including
ResNet-18, ResNet-20, and ResNet-32, and datasets like CIFAR-10, CIFAR-100, and
ImageNet (2012), showcasing its applicability in computer vision. Our
experiments show that the proposed method can yield substantial storage savings
while maintaining or even enhancing classification performance.