Towards Explaining Deep Neural Network Compression Through a
  Probabilistic Latent Space
Mahsa Mozafari-Nia Salimeh Yasaei Sekeh
Despite the impressive performance of deep neural networks (DNNs), their
computational complexity and storage space consumption have led to the concept
of network compression. While DNN compression techniques such as pruning and
low-rank decomposition have been extensively studied, there has been
insufficient attention paid to their theoretical explanation. In this paper, we
propose a novel theoretical framework that leverages a probabilistic latent
space of DNN weights and explains the optimal network sparsity by using the
information-theoretic divergence measures. We introduce new analogous projected
patterns (AP2) and analogous-in-probability projected patterns (AP3) notions
for DNNs and prove that there exists a relationship between AP3/AP2 property of
layers in the network and its performance. Further, we provide a theoretical
analysis that explains the training process of the compressed network. The
theoretical results are empirically validated through experiments conducted on
standard pre-trained benchmarks, including AlexNet, ResNet50, and VGG16, using
CIFAR10 and CIFAR100 datasets. Through our experiments, we highlight the
relationship of AP3 and AP2 properties with fine-tuning pruned DNNs and
sparsity levels.