Deep convolutional neural network (DCNN) based supervised learning is a widely practiced approach for large-scale image classification.   However, retraining these large networks to accommodate new, previously unseen data demands high computational time and energy requirements. Also, previously seen training samples may not be available at the time of retraining. We propose an efficient training methodology and incrementally growing a DCNN to allow new classes to be learned while sharing part of the base network. Our proposed methodology is inspired by transfer learning techniques, although it does not forget previously learned classes. An updated network for learning new set of classes is formed using previously learned convolutional layers (shared from initial part of base network) with addition of few newly added convolutional kernels included in the later layers of the network. We evaluated the proposed scheme on several recognition applications. The classification accuracy achieved by our approach is comparable to the regular incremental learning approach (where networks are updated with new training samples only, without any network sharing). Deep Convolutional Neural Networks (DCNNs) have achieved remarkable success in various cognitive applications, particularly in computer vision BID14 . They have shown human like performance on a variety of recognition, classification and inference tasks, albeit at a much higher energy consumption. One of the major challenges for convolutional networks is the computational complexity and the time needed to train large networks. Since training of DCNNs requires state-ofthe-art accelerators like GPUs BID0 , large training overhead has restricted the usage of DCNNs to clouds and servers. It is common to pre-train a DCNN on a large dataset (e.g. ImageNet, which contains 1.2 million images with 1000 categories), and then use the trained network either as an initialization or a fixed feature extractor for the specific application BID16 . A major downside of such DCNNs is the inability to learn new information since the learning process is static and only done once before it is exposed to practical applications. In real-world scenarios, classes and their associated labeled data are always collected in an incremental manner. To ensure applicability of DCNNs in such cases, the learning process needs to be continuous. However, retraining these large networks using both previously seen and unseen data to accommodate new data, is not feasible most of the time. Incremental learning plays a critical role in alleviating this issue by ensuring continuity in the learning process through regular model update based only on the new available batch of data. A system that can learn incrementally is beneficial in practical situations, since it can gradually expand its capacity to accommodate increasing number of classes. Nevertheless, incremental learning can be computationally expensive and time consuming, if the network is large enough. This paper focuses on incremental learning of deep convolutional neural network (DCNN) for image classification task. In doing so, we attempt to address the more fundamental issue: an efficient learning system must deal with new knowledge that it is exposed to, as humans do. To achieve this goal, there are two major challenges. First, as new data becomes available, we should not start learning from scratch. Rather, we leverage what we have already learned and combine them with new knowledge in a continuous manner. Second, to accommodate new data, if there is a need to increase the capacity of our network, we will have to do it in an efficient way.There have been several prior works on incremental learning of neural networks. Many of them focus on learning new classes from fewer samples BID2 BID6 ) utilizing transfer learning techniques. To avoid learning new categories from scratch, BID2 proposed a Bayesian transfer learning method using very few training samples. By introducing attribute-based classification the authors BID6 ) achieved zero-shot learning (learning a new class from zero samples). These works rely on shallow models instead of DCNN, and the category size is small in comparison. The challenge of applying incremental learning (transfer learning as well) on DCNN lies in the fact that it consists of both feature extractor and classifier in one architecture. BID13 utilized ensemble of classifiers by generating multiple hypotheses using training data sampled according to carefully tailored distributions. The outputs of the resulting classifiers are combined using a weighted majority voting procedure. Inspired form BID13 , BID8 utilized ensemble of modified convolutional neural networks as classifiers by generating multiple hypotheses. The existing classifiers are improved in BID13 BID8 ) by combining new hypothesis generated from newly available examples without compromising classification performance on old data. The new data in BID13 BID8 ) may or may not contain new classes. BID19 proposed a training algorithm that grows a network not only incrementally but also hierarchically. Classes are grouped according to similarities, and self-organized into different levels of the hierarchy. All new networks are cloned from existing ones and therefore inherit learned features. These new networks are fully retrained and connected to base network. The problem with this method is the increase of hierarchical levels as new set of classes are added over time.Our work differs in goal, as we want to grow a DCNN to accommodate new set of classes by network sharing, without forgetting the old classes. For learning new set of classes, we form a new network by reusing previously learned convolutional layers (shared from initial part of the base network) followed by new (added) trainable convolutional layers towards the later layers of the network. The shared convolutional layers work as fixed feature extractors (learning parameters are frozen) and minimize learning overhead for new set of classes. The error resilience property of neural network ensures that even if we freeze some of the learning parameters, the network will be able to adapt to it and learn. In fact, utilizing this attribute, in recent work BID15 showed the use of fixed Gabor filters as convolutional kernels in conjunction with regular trainable convolutional kernels. Since the frozen parameters are from an already learned network of similar (not same) classes, it further guarantees that the new classes can be learned smoothly without having convergence issues. By sharing initial convolutional layers of the base network, a significant fraction of the powerhungry components of the backpropagation training was eliminated, thereby achieving considerable reduction in training energy while maintaining competitive output accuracy.The novelty of this work lies in the fact that we developed an empirical mechanism to identify how much of the network can be shared as new classes are added. In this work, we also quantified the energy consumption, training time and memory storage savings associated with models trained with different amounts of sharing to emphasize the importance of network sharing from hardware point of view. In summary, the key contributions of our work are as follows:• We propose sharing of convolutional layers to reduce computational complexity while training a network to accommodate new set of classes.• We developed a methodology to identify optimal sharing of convolutional layers in order to get the best trade-off between accuracy and other parameters of interest, especially computation energy consumption, training time and memory access.• We developed an energy model for quantifying energy consumption of the network during training, based on the Multiplication and Accumulation (MAC) operations in the training algorithm.• We substantiate the scalability and robustness of the proposed methodology by applying the proposed method to different network structures trained for different benchmark datasets.We show that our proposed methodology leads to energy efficiency, reduction in storage requirements, memory access and training time, while maintaining classification accuracy. The performance of DCNNs relies greatly on the availability of a representative set of the training examples. Generally, in practical applications, data acquisition and learning process is time consuming. Also, it is very likely that the data are available in small batches over a period of time. A competent classifier should be able to support an incremental method of accommodating new data without losing ground on old data inference capability. In this paper, we introduce an incremental training methodology for DCNNs, which employs partial network sharing. This method allows us to accommodate new, previously unseen data without the need of retraining the whole network with previously seen data. It can preserve existing knowledge, and can accommodate new information. Importantly, all new networks start from an existing base network and share learning parameters. The new updated network inherit features from the base network by sharing convolutional layers, leading to improved computational effort and energy consumption during training and thus, speed up the learning process. We applied the proposed method on different DCNNs trained on real-world recognition applications. Results confirm the scalability of the proposed approach with significant improvements.
The paper is about a new energy-efficient methodology for Incremental learning Proposes procedure for incremental learning as transfer learning. The paper presents a method to train deep convolutional neural networks in an incremental fashion, in which data are available in small batches over a period of time. Presents an approach to class-incremental learning using deep networks by proposing three different learning strategies in the final/best approach.