Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint. We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer. Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio. Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation. Despite decades of research, the design of neural networks is still an empirical process. Practitioners make design choices, such as the number of layers, type of layers, number of filters per layer, etc., based on intuition or brute-force search. Nevertheless, the performance of these algorithms, together with the advances of GPU devices, have led to a growing popularity of these techniques in both academia and industry. Recent advances are unveiling some properties of neural networks. For example, there is a consensus that depth can accelerate learning, and that wider layers help optimization BID2 BID29 . However, in practical applications, the size of these networks is often a limiting factor when deploying on devices with constrained storage, memory, and computation resources.Another known neural network property is that the responses of a layer exhibit considerable correlation BID12 , inspiring the idea of learning decorrelated filters BID8 BID38 . These algorithms propose a modified loss function to encourage decorrelation during training and show that accuracy improves with decorrelated filters. However, such algorithms focus on training and do not address network compression. Our hypothesis is that layers that exhibit high correlation in filter responses could learn equally well using a smaller number of filters.Principal Filter Analysis (PFA) draws from the recent findings that it is easier to start with an overparametrized network and it then exploits intra-layer correlation for guiding network compression. PFA analyzes a trained network and is agnostic to the training methodology and the loss function. Inference is performed on a dataset, and the correlation within the responses of each layer is used to provide a compression recipe. A new smaller architecture based on this recipe can then be retrained.We propose two closed-form algorithms based on spectral energy analysis for suggesting the number of filters to remove in a layer:PFA-En uses Principal Component Analysis (PCA) BID21 to allow a user to specify the proportion of the energy in the original response that should be preserved in each layer; PFA-KL is a heuristic that leads to a parameter-free approach that uses Kullback-Leibler (KL) divergence BID24 to identify the number of redundant filters.Based on the new suggested number of filters per layer identified by PFA, we remove those that are maximally correlated with other filters and fine-tune the network by retraining. Both PFA algorithms are straightforward to implement and, as shown in Sec. 4, they achieve better compression and, in most cases, better accuracy than the state of the art on several datasets and architectures. In Sec. 4 we also show how PFA can be used to perform simulations compression and domain adaptation. Two effective, and yet easy to implement techniques for the compression of neural networks using Principal Filter Analysis were presented: PFA-En and PFA-KL. These techniques exploit the inherent correlation of filter responses within layers to compress networks without compromising accuracy. These techniques can be applied to the output response of any layer with no knowledge of the training procedure or the loss function. While easy to implement, both algorithms surpass state of the art results in terms or compression ratio with the advantage that PFA-KL is parameter free and PFA-En has only a single intuitive parameter: the energy to be preserved in each layer or a desired network characteristic (such as a target footprint or FLOPs).By performing spectral analysis on the responses rather than the weights, PFA allows users to take advantage of transfer learning in order to perform domain adaptation and derive smaller specialized networks that are optimally designed for a specific task, while starting from the same base model.It is interesting to observe that all compression algorithms in the state of art, including PFA-KL, do not converge if applied repeatedly. This is due to the retraining step which alters the weights in order to optimize an objective function that does not take into account compression criteria. In practical applications this is not a limitation since most of the models tend to be over specified and users can apply compression algorithms until the accuracy falls below an acceptable level. However, it is an interesting theoretical limitation that could inspire future work. Specifically for PFA, this could be a strategy that can also suggest expanding layers so that an optimal size could be reached at convergence.
We propose an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprints. Proposes to prune convolutional networks by analyzing the observed correlation between the filters of a same layer as expressed by the eigenvalue spectrum of their covariance matrix. This paper introduces an approach to compressing neural networks by looking at the correlation of filter responses in each layer via two strategies. This paper proposes a compression method based on spectral analysis