Tensor network compressibility of convolutional models
Sukhbinder Singh Saeed S. Jahromi Roman Orus
Convolutional neural networks (CNNs) represent one of the most widely used
neural network architectures, showcasing state-of-the-art performance in
computer vision tasks. Although larger CNNs generally exhibit higher accuracy,
their size can be effectively reduced by "tensorization" while maintaining
accuracy. Tensorization consists of replacing the convolution kernels with
compact decompositions such as Tucker, Canonical Polyadic decompositions, or
quantum-inspired decompositions such as matrix product states, and directly
training the factors in the decompositions to bias the learning towards
low-rank decompositions. But why doesn't tensorization seem to impact the
accuracy adversely? We explore this by assessing how truncating the convolution
kernels of dense (untensorized) CNNs impact their accuracy. Specifically, we
truncated the kernels of (i) a vanilla four-layer CNN and (ii) ResNet-50
pre-trained for image classification on CIFAR-10 and CIFAR-100 datasets. We
found that kernels (especially those inside deeper layers) could often be
truncated along several cuts resulting in significant loss in kernel norm but
not in classification accuracy. This suggests that such ``correlation
compression'' (underlying tensorization) is an intrinsic feature of how
information is encoded in dense CNNs. We also found that aggressively truncated
models could often recover the pre-truncation accuracy after only a few epochs
of re-training, suggesting that compressing the internal correlations of
convolution layers does not often transport the model to a worse minimum. Our
results can be applied to tensorize and compress CNN models more effectively.