Modern Convolutional Neural Networks (CNNs) are complex, encompassing millions of parameters. Their deployment exerts computational, storage and energy demands, particularly on embedded platforms. Existing approaches to prune or sparsify CNNs require retraining to maintain inference accuracy. Such retraining is not feasible in some contexts. In this paper, we explore the sparsification of CNNs by proposing three model-independent methods. Our methods are applied on-the-fly and require no retraining. We show that the state-of-the-art models' weights can be reduced by up to 73% (compression factor of 3.7x) without incurring more than 5% loss in Top-5 accuracy. Additional fine-tuning gains only 8% in sparsity, which indicates that our fast on-the-fly methods are effective. There has been a significant growth in the number of parameters (i.e., layer weights), and the corresponding number of multiply-accumulate operations (MACs), in state-of-the-art CNNs BID14 BID13 BID19 BID23 BID8 BID11 BID24 BID22 . Thus, it is to no surprise that several techniques exist for "pruning" or "sparsifying" CNNs (i.e., forcing some model weights to 0) to both compress the model and to save computations during inference. Examples of these techniques include: iterative pruning and retraining BID2 BID7 BID3 BID20 BID17 , Huffman coding BID5 , exploiting granularity BID15 BID4 , structural pruning of network connections BID25 BID16 BID0 BID18 , and Knowledge Distillation (KD) BID9 .A common theme to the aforementioned techniques is that they require a retraining of the model to fine-tune the remaining non-zero weights and maintain inference accuracy. Such retraining, while feasible in some contexts, is not feasible in others, particularly industrial ones. For example, for mobile platforms, a machine learning model is typically embedded within an app for the platform that the user directly downloads. The app utilizes the vendor's platform runtime support (often in the form of a library) to load and use the model. Thus , the platform vendor must sparsify the model at runtime, i.e., on-the-fly, within the library with no opportunity to retrain the model. Further , the vendor rarely has access to the labelled data used to train the model. While techniques such as Knowledge Distillation BID9 can address this lack of access, it is not possible to apply it on-the-fly.In this paper, we develop fast retraining-free sparsification methods that can be deployed for on-thefly sparsification of CNNs in the contexts described above. There is an inherent trade-off between sparsity and inference accuracy. Our goal is to develop model-independent methods that result in large sparsity with little loss to inference accuracy. We develop three model-independent sparsification methods: flat, triangular, and relative. We implement these methods in TensorFlow and use the framework to evaluate the sparsification of several pretrained models: Inception-v3, MobileNet-v1, ResNet, VGG, and AlexNet. Our evaluation shows that up to 81% of layer weights in some models may be forced to 0, incurring only a 5% loss in inference accuracy. While the relative method appears to be more effective for some models, the triangular method is more effective for others. Thus, a predictive modeling autotuning BID6 BID1 is needed to identify, at run-time, the optimal choice of method and it hyper-parameters.
In this paper, we develop fast retraining-free  sparsification methods that can be deployed for on-the-fly sparsification of CNNs in many industrial contexts. This paper proposes approaches for pruning CNNs without retraining by introducing three schemes to determine the thresholds of pruning weights. This paper describes a method for sparsification of CNNs without retraining.