Although deep neural networks show their extraordinary power in various tasks, they are not feasible for deploying such large models on embedded systems due to high computational cost and storage space limitation. The recent work knowledge distillation (KD) aims at transferring model knowledge from a well-trained teacher model to a small and fast student model which can significantly help extending the usage of large deep neural networks on portable platform. In this paper, we show that, by properly defining the neuron manifold of deep neuron network (DNN), we can significantly improve the performance of student DNN networks through approximating neuron manifold of powerful teacher network. To make this, we propose several novel methods for learning neuron manifold from DNN model. Empowered with neuron manifold knowledge, our experiments show the great improvement across a variety of DNN architectures and training data. Compared with other KD methods, our Neuron Manifold Transfer (NMT) has best transfer ability of the learned features. In recent years, deep neural networks become more and more popular in computer vision and neural language processing. A well-trained learning model shows its power on tasks such as image classification, object detection, pattern recognizing, live stream analyzing, etc. We also have the promise that given enough data, deeper and wider neural networks can achieve better performance than the shallow networks BID0 ). However, these larger but well-trained networks also bring in high computational cost, and leave large amount of memory footprints which make these models very hard to travel and reproduce BID7 ). Due to this drawback, a massive amount of trainable data gathered by small devices such as mobiles, cameras, smart sensors, etc. is unable to be utilized in the local environment which can cause time-sensitive prediction delay and other impractical issues.To address the above issues, recently, there are extensive works proposed to mitigate the problem of model compression to reduce the computational burden on embedded system. Back to the date 2006, Buciluǎ et al. first proposed to train a neural network to mimic the output of a complex and large ensemble. This method uses ensemble to label the unlabeled data and trains the neural network with the data labeled by the ensemble, thus mimicking the function which learned by the ensemble and achieves similar accuracy. Based on the idea of (Buciluǎ,Geoffrey et al.) originally introduced a student-teacher paradigm in transferring the knowledge from a deeper and wider network (teacher) to a shallow network (student). They call this student-teacher paradigm as knowledge distillation (KD). By properly defining the knowledge of teacher as softened softmax (soft target), the student learns to mimic soft target distribution for each class. Thanks to Hinton's pioneer work, a series of subsequent works have sprung up by utilizing different forms of knowledge. BID22 regard the spatial attention maps of a convolution neural network as network knowledge. However, an implicit assumption that they make is that the absolute value of a hidden neuron activation can be used as an indication about the importance of that neuron w.r.t. the specific input which limited their application only fit for image classification tasks. Another assumption that has been widely used is from BID0 that deeper networks always learn better representation. Based on that, FitNets BID13 ) tries to learn a thin deep network using a shallow one with more parameters. They believe that the convolution regressor is the network knowledge which can inherit from the teacher to its student. In 2017, researcher from TuSimple ( TuSimple et al. (2015) , using softmax as knowledge, the student network mimics teacher's softmax and minimize the loss on soft target. Middle one, mentioned in BID22 , named as attention transfer, an additional regularizer has been applied known as attention map, student needs to learn the attention map and soft target. The right part is our neuron manifold transfer, where we take neuron manifold as knowledge, and it reduces the computational and space cost.(2015 )) introduces two new definitions of network knowledge, BID8 takes the advantage of Maximum Mean Discrepancy (MMD) to minimize the distance metric in probability distributions, and they regard network knowledge as class distribution. BID4 propose to transfer the cross sample similarities between the student and teacher to improve the performance of transferred networks. We notice that in the DarkRank the network knowledge is defined as cross sample similarities.By reviewing extensive KD works, we notice that the key point in knowledge transfer is how we define the network knowledge, and in fact, a well-defined network knowledge can greatly improve the performance of the distilled network. Moreover, in our perspective, a perfect knowledge transfer method must allow us to transfer one neural network architecture into another, while preserving other generalization. A perfect transfer method, however, would use little observations to train, optimally use the limited samples at its disposal. Unfortunately , to our best knowledge, due to the complexity of large DNN, simply mimicking the teacher logit or a part of teacher features properties is far away to be benefited. Therefore, if we look back and consider the essence of DNN training, we notice that, another point of view to look at the distribution of neuron features is the shape of that feature. Here, the shape of neuron features include the actual value and the relative distance between two features. That is, during the process of knowledge transfer, student network not only learns the numerical information but also inherits the geometric properties. Therefore, in order to track the change of large DNN feature knowledge, a manifold approximation technique is vying for our attention. Manifold learning has been widely used in Topological Data Analysis (TDA) BID3 ), and this technique can project the high dimensional data to a lower dimensional manifold and preserving both numerical feature properties and geometric properties. In previous works, feature mapping causes computational resource waste, and class distribution matching is limited the usage. However, using the neuron manifold information, we not only collect as much as possible feature properties which can greatly represent feature, but also preserve inter-neuron characteristics (spatial relation). Since manifold projection can greatly reduce the dimension of teacher feature, we compress the teacher model and make student model more reliable. To summarize, the contributions of this paper are three folds:• We introduce a new type of knowledge that is the d-dimensional smooth sub-manifold of teacher feature maps called neuron manifold.• We formalize manifold space in feature map, and implement in details.• We test our proposed method on various metric learning tasks. Our method can significantly improve the performance of student networks. And it can be applied jointly with existing methods for a better transferring performance.We test our method on MNIST, CIFAR-10, and CIFAR-100 datasets and show that our Neuron Manifold Transfer (NMT) improves the students performance notably. In this paper, we propose a novel method for knowledge transfer and we define a new type network knowledge named neuron manifold. By utilizing the state of art technique in Topological Data Analysis, we extract the DNN's feature properties and its geometric properties. We test our NMT on various dataset and the results are quite promising, thus further confirming that our knowledge transfer method could indeed learn better feature representations. They can be successfully transferred to high level vision task in the future. We believe that our novel view will facilitate the further design of knowledge transfer methods. In our future work, we plan to explore more applications of our NMT methods, especially in various regression problems, such as super resolution and optical flow prediction, etc.
A new knowledge distill method for transfer learning The work introduces a knowledge distillation method using the proposed neuron manifold concept.  Proposes a knowledge distilling method in which neural manifold is taken as the transferred knowledge.