Towards Generalized Entropic Sparsification for Convolutional Neural
  Networks
Tin Barisin Illia Horenko
Convolutional neural networks (CNNs) are reported to be overparametrized. The
search for optimal (minimal) and sufficient architecture is an NP-hard problem
as the hyperparameter space for possible network configurations is vast. Here,
we introduce a layer-by-layer data-driven pruning method based on the
mathematical idea aiming at a computationally-scalable entropic relaxation of
the pruning problem. The sparse subnetwork is found from the pre-trained (full)
CNN using the network entropy minimization as a sparsity constraint. This
allows deploying a numerically scalable algorithm with a sublinear scaling
cost. The method is validated on several benchmarks (architectures): (i) MNIST
(LeNet) with sparsity 55%-84% and loss in accuracy 0.1%-0.5%, and (ii) CIFAR-10
(VGG-16, ResNet18) with sparsity 73-89% and loss in accuracy 0.1%-0.5%.