Deep network compression seeks to reduce the number of parameters in the network while maintaining a certain level of performance.   Deep network distillation seeks to train a smaller network that matches soft-max performance of a larger network.   While both regimes have led to impressive performance for their respective goals, neither provide insight into the importance of a given layer in the original model, which is useful if we are to improve our understanding of these highly parameterized models.   In this paper, we present the concept of deep net triage, which individually assesses small blocks of convolution layers to understand their collective contribution to the overall performance, which we call \emph{criticality}.  We call it triage because we assess this criticality by answering the question: what is the impact to the health of the overall network if we compress a block of layers into a single layer.
 We propose a suite of triage methods and compare them on problem spaces of varying complexity.   We ultimately show that, across these problem spaces, deep net triage is able to indicate the of relative importance of different layers.   Surprisingly, our local structural compression technique also leads to an improvement in overall accuracy when the final model is fine-tuned globally. As computational devices and methods become more powerful, deep learning models are able to grow ever deeper BID16 . To grow so deep, the most modern of networks have relied on clever intermediary layers-such as shortcut connection layers in BID5 -to overcome overfitting. While these methods allow for learning representations afforded only by very deep architectures, it is known that there are still more extraneous features and connections learned by these networks BID13 . The question of how to best remove these redundancies has been the focus of deep compression methods BID13 BID2 BID4 BID9 . Still others have investigated the ability of smaller-difficult to trainnetworks to learn from parent models via a method known as Knowledge Distillation BID6 BID14 .Both of these classes of approaches have demonstrated impressive performance for their respective goals; essentially, both lead to smaller networks that can match the performance of their larger, parent network. This performance is achieved in a variety of ways. For example, BID4 reduces the number of network parameters via low-threshold pruning, followed by retraining, weight quantization and sharing, in tandem with low-rank approximations to ensure removal of redundant and unimportant weights. These methods can be thought of as finding a sparser, compressed model which best approximates the original.Similarly, knowledge distillation methods leverage the soft-max outputs of previously trained "teacher" networks and network ensembles as guidance to train a smaller network, which would have otherwise been too difficult to train BID6 BID14 BID15 . These knowledge distillation networks globally train the smaller network to best approximate the soft-max output of the original network, sometimes with per-layer, intermediary targets incorporated BID0 BID11 BID17 .While impressive, these two methods do not shed any light on the criticality, or the relative importance of a given layer or block of layers to the overall output. Such layer-based analysis is important to understanding these increasingly deep networks, even if only in an empirical sense.To that end, we propose an idea called deep net triage that independently assesses small blocks of layers with respect to their importance to the overall network health. We drive the triage by using the initial parent network as an initialization, like BID1 , rather than as a means of globally retraining. Triage works by removing a connected block of network layers and replacing them with a single layer; we focus on convolution layers in this paper. We iterate over all connected blocks of layers separately thereby assessing the role each set plays in the original parent network.Our means for this triage is local structural compression, which approximates a section of a disassembled network and assesses the ability to approximate and relearn the original model. With structural compression we compress segments of a deep network-VGG16-and attempt to recover the compressed layer of the network via various initialization and training methodologies BID16 . We structure this as a compression problem as we are approximating multiple convolutional layers' representational abilities with a single, selectively initialized and trained layer. Distinctly, though , our primary goal is not to seek maximal compressive performance. Rather we seek to investigate the robustness of a network when faced with structural alterations, and how various learning techniques affect this behavior across data sets of assorted complexity. We seek overarching trends between these methods, layers, and data sets in hopes of developing a greater understanding for the representational ability, and robustness of neural networks.We perform our analysis using five approaches to structural compression for deep net triage, and four different data sets. We find that of the five approaches, methods which fine-tune over the entirety of the network achieve best performance across all data sets. Furthermore, these fine-tuned models are able to match or even exceed the performance of the baseline model. This suggests that for superior performance, a network cannot be altered without again retraining over the entire network. We additionally demonstrate that the criticality of any single layer in the network is not sufficient to inhibit relearning of the representations of the parent network. Finally, we show that knowledge distillation is an effective means of transferring the learned representations from a teacher to a student at any given intermediate point, even when the layer is altered or compressed, and improves a model's convergence. We present a novel method of analyzing deep learning methods which we refer to as deep net triage. By drawing from the field of deep network compression and knowledge distillation we design experiments which question the criticality of layers within a network structure, and assess the representations learned therein. We structurally compress a layer at a time, while conducting a series of experiments across these layers on various data sets to infer about the layer's ability to learn representations, recover from compression, and integrate itself into the global network.We show through experimentation that structurally compressed and fine-tuned models can perform equivalent to, or better than parent, uncompressed models in a layer invariant manner. Additionally, we show that parent-inspired initialization regimes applied only at the layer are unable to compete with fine-tuning over the entire global model. Lastly, we show that Student-Teacher models evaluated at intermediate layers in the form of hints from uncompressed parent models promote faster convergence to maximal accuracies, despite being unable to outperform full model training methods.Through this work, we hope to spur others to devise and rigorously test targeted assessments of deep networks, as we do in our deep net triage. While, as a community, we may continue to develop ever better performing methods for given problem spaces, we will never truly advance as a field until further intuition for and understanding of deep networks is developed. As optimization and statistical theory progresses on one side, so too must experimentalists approach from the other.
We seek to understand learned representations in compressed networks via an experimental regime we call deep net triage Compares various initialization and training methods of transferring knowledge from VGG network to a smaller student network by replacing blocks of layers with single layers. This paper presents five methods for doing triaging or block layer compression for deep networks. The paper proposes a method to compress a block of layers in a NN that evaluates several different sub-approaches