This paper presents a storage-efficient learning model titled Recursive Binary Neural Networks for embedded and mobile devices having a limited amount of on-chip data storage such as hundreds of kilo-Bytes. The main idea of the proposed model is to recursively recycle data storage of weights (parameters) during training. This enables a device with a given storage constraint to train and instantiate a neural network classifier with a larger number of weights on a chip, achieving better classification accuracy. Such efficient use of on-chip storage reduces off-chip storage accesses, improving energy-efficiency and speed of training. We verified the proposed training model with deep and convolutional neural network classifiers on the MNIST and voice activity detection benchmarks. For the deep neural network, our model achieves data storage requirement of as low as 2 bits/weight, whereas the conventional binary neural network learning models require data storage of 8 to 32 bits/weight. With the same amount of data storage, our model can train a bigger network having more weights, achieving 1% less test error than the conventional binary neural network learning model. To achieve the similar classification error, the conventional binary neural network model requires 4× more data storage for weights than our proposed model. For the convolution neural network classifier, the proposed model achieves 2.4% less test error for the same on-chip storage or 6× storage savings to achieve the similar accuracy.
 Deep Neural Networks (DNN) have demonstrated the state-of-the-art results in a wide range of cognitive workloads such as computer vision BID10 and speech recognition ), achieving better-than-human performance for the tasks often considered too complex for machines. The success of DNNs has indeed motivated scientists and engineers to implement a DNN in mobile and embedded devices, dubbed as Internet of Smart Things BID9 ). The recent works in this area, however, mostly implement the inference function of DNN, rather than training, while training is performed in cloud computers and posttraining weights are downloaded to mobile and embedded devices BID11 ).On-device learning, however, becomes increasingly important for the mobile and embedded devices for the following three reasons. First, an intelligent device benefits to have the model that is custombuilt for the device itself, its end user, and environment. This is because the model tends to be more accurate and effective if constructed with the consideration of those factors. Second, the training data from mobile and embedded devices can contain security-sensitive information, e.g., personal health data from wearable medical devices. At the risk of being leaked, users typically do not want to upload such data onto cloud computers. Finally, in the era of Internet of Things (IoT), we anticipate a drastic increase in the number of deployed devices, which can proportionally increase the number of learning tasks to be done in the cloud. Coupled with the complexity of training, even for powerful cloud computers, this can be a computationally challenging task.On-device learning, however, entails various challenges in algorithms, data, and systems BID15 ; BID18 ). The most eminent challenge regarding computing systems is high energy consumption caused by dense computation and data access, which is considered prohibitive for the limited resources of embedded devices. The high overhead of data access is caused by fetching DNN weights from DRAM (or FLASH) external to a computing chip on an embedded device. Since the data storage size is limited for such computing chip, the parameters of a DNN have to be stored in external DRAM and FLASH during training. For example, ARM Cortex M3 processor, a processor widely used in commercial wearable devices such as FitBit, has only 64 kilo-Byte (kB) on-chip data storage. This can only store very small size of DNN especially if each weight is 32-bit float point number. Compared to accessing on-chip SRAM, accessing off-chip DRAM incurs 3 to 4 orders of magnitudes more energy and delay overhead. Therefore, fetching weights every time for each data makes training prohibitive to be implemented on a mobile and embedded device BID5 ).Recently several techniques such as pruning, distilling, and binarizing weights have been proposed to compress the parameters of a DNN. This makes it more feasible to fit weights in on-chip SRAM BID5 ; BID2 ; BID14 ; BID8 ). These techniques can also reduce computation overhead. However, these works focused on weight size compression after training is finished. The data storage requirement during training remains the same.Similarly, several learning models, which belong to so-called Binary Neural Networks (BNN), have been proposed BID2 ; BID14 ). These model uses sign bits (or binary information ) of weights in several parts of the learning model notably the part of multiplying and accumulating weights with inputs/activations. Although this greatly reduces computational complexity , each weight still needs to be represented in high precision number with multiple bits (e.g. 32 bits in BID2 ; BID14 ) during the end-to-end training process. This is because weights have to be fine-tuned in the weight update part. Therefore, this so-called BNN models have not demonstrated to scale storage requirement for training below 32 bits/weight.Our goal is, therefore, to efficiently use the limited amount of on-chip data storage during training. We also aim to scale computational complexity. Toward this goal, we propose a new learning model, Recursive Binary Neural Network (RBNN). This model is based on the process of weight training, weight binarization, recycling storage of the non-sign-bit portion of weights to add more weights to enlarge the neural network for accuracy improvement. We recursively perform this process until either accuracy stops improving or we use up all the storage on a chip.We verified the proposed RBNN model on a Multi-Layer Perceptron (MLP)-like and a convolutional neural network classifier on the MNIST and Voice Activity Detection (VAD) benchmark. We considered typical storage constraints of embedded sensing devices in the order of hundreds of kB. The experiment in the MLP-like classifier on MNIST confirms that the proposed model (i) demonstrates 1% less test error over the conventional BNN learning model specifically following BID2 for the same storage constraints or (ii) scales on-chip data storage requirement by 4× for the same classification test error rate(∼2%), marking the storage requirement of 2 bits/weight. The conventional BNN models in BID2 ; BID14 exhibit a significantly larger storage requirements of 8 to 32 bits/weight. The experiment of the CNN classifier for MNIST confirms up to 6× reduction of data storage requirement and 2.4% less test error. For the VAD benchmark, the proposed RBNN achieves 9× savings in data storage requirement .The remainder of the paper is as follow. In Sec. 2 we will introduce the works related to this paper, including comparison to existing works on distillation, compression, BNNs, and low-precision weights. In Sec. 3 we will describe our proposed model. Sec. 4 will present the experimental results and comparisons to the conventional BNN model. Finally , in Sec. 5, we will conclude the paper. The paper includes Appendix A to D to describe additional experiments and analysis.2 RELATED WORK 2.1 DISTILLATION AND COMPRESSION OF DNN PARAMETERS Knowledge distillation BID8 ) is a technique to compress knowledge of an ensemble of DNNs into one small DNN while maintaining the accuracy. Although this technique can scale the number of weights for deployment systems post-training, it cannot scale data storage requirement Under review as a conference paper at ICLR 2018 for training. Specifically, during training, each of weights is represented in high-precision number, which needs to be stored in multi-bit data storage.Another technique is to compress the data size of weights by exploiting redundancies in them. In BID5 , the authors combine four sub-techniques, namely weight pruning, quantization, sharing, and compression coding to reduce the data size of weights. Similar to the knowledge distillation, this technique can be applied to the weights that are already trained, and cannot scale data storage requirement of weights during training. This paper presents a new learning model for on-device training with limited data storage. The proposed RBNN model efficiently uses limited on-chip data storage resources by recycling the part of data storage that would have been wasted in conventional BNN model, to add and train more weights to a neural network classifier. We verified the proposed model with MLP-like DNN and CNN classifiers on the MNIST and VAD benchmark under the typical embedded device storage constraints. The results of MLP-like DNNs on MNIST show that the proposed model achieves 2 bits/weight storage requirement while achieving 1% less test error as compared to the conventional BNN model for the same storage constraint. Our proposed model also achieves 4× less data storage than the conventional model for the same classification error. The similar to greater savings are verified with the CNN classifiers and the VAD benchmarks. We expect the future work of further reduce computation complexity, such as binarization of activation function of BNN BID3 ). We also expect to apply the RBNN model to the ensembles of neural networks BID20 , and the mixture of experts BID16 ).
We propose a learning model enabling DNN to learn with only 2 bit/weight, which is especially useful for on-device learning Proposes a method to discretisize a NN incrementally to improve memory and performance.