Deep learning models have outperformed traditional methods in many fields such
 as natural language processing and computer vision. However, despite their
 tremendous success, the methods of designing optimal Convolutional Neural Networks
 (CNNs) are still based on heuristics or grid search. The resulting networks
 obtained using these techniques are often overparametrized with huge computational
 and memory requirements. This paper focuses on a structured, explainable
 approach towards optimal model design that maximizes accuracy while keeping
 computational costs tractable. We propose a single-shot analysis of a trained CNN
 that uses Principal Component Analysis (PCA) to determine the number of filters
 that are doing significant transformations per layer, without the need for retraining.
 It can be interpreted as identifying the dimensionality of the hypothesis space
 under consideration. The proposed technique also helps estimate an optimal number
 of layers by looking at the expansion of dimensions as the model gets deeper.
 This analysis can be used to design an optimal structure of a given network on
 a dataset, or help to adapt a predesigned network on a new dataset. We demonstrate
 these techniques by optimizing VGG and AlexNet networks on CIFAR-10,
 CIFAR-100 and ImageNet datasets. This analysis has only been done on activation outputs for convolutional layers before the application of non-linearities such as ReLU. Non-linearities introduce more dimensions, but those are not a function of the number of filters in a layer. Hence we recommend not to perform ReLU in-place while performing this analysis. The number of samples to be taken into account for PCA are recommended to be around 2 orders of magnitudes more than the number of filters we are trying to find redundancy in. This is particularly of importance in the later layers where the activation map sizes are small. We need to collect these activations over many batches to make sure we have enough data to run PCA analysis on. While the percentage variance one would like to retain depends on the application and acceptable error tolerance, empirically we have found that preserving 99.9% puts us at a sweet spot for most cases with less than half a percentage point in accuracy degradation and a considerable gain in computational cost. This analysis comes in handy in three cases: While designing new network for new data; while adapting given network for new data; and while optimizing current network for faster runtimes or reduced power consumption during training or inference in hardware implementations. Another benefit of this analysis is that not only does it deliver an optimal point, but enables an interpretable, graceful exploration of accuracy-energy trade-off with negligible overhead of compute cost and time. This method is orthogonal to other model compression techniques.
We present a single shot analysis of a trained neural network to remove redundancy and identify optimal network structure This paper proposes a set of heuristics for identifying a good neural network architecture, based on PCA of unit activations over the dataset This paper presents a framework for optimising neural networks architectures through the identification of redundant filters across layers