In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch.
 When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task.
 However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task.
 In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model.
 We eventually recommend a simple $L^2$ penalty using the pre-trained model as a reference, and we show that this approach behaves much better than the standard scheme using weight decay on a partially frozen network. It is now well known that modern convolutional neural networks (e.g. BID14 BID25 BID11 BID28 ) can achieve remarkable performance on large-scale image databases, e.g. ImageNet BID3 ) and Places 365 BID37 ), but it is really dissatisfying to see the vast amounts of data, computing time and power consumption that are necessary to train deep networks. Fortunately, such convolutional networks, once trained on a large database, can be refined to solve related but different visual tasks by means of transfer learning, using fine-tuning BID34 BID25 .Some form of knowledge is believed to be extracted by learning from the large-scale database of the source task and this knowledge is then transferred to the target task by initializing the network with the pre-trained parameters. However , after fine-tuning, some of the parameters may be quite different from their initial values, resulting in possible losses of general knowledge that may be relevant for the targeted problem. In particular , during fine-tuning, L 2 regularization drives the parameters towards the origin and thereby encourages large deviations between the parameters and their initial values.In order to help preserve the acquired knowledge embedded in the initial network, we consider using other parameter regularization methods during fine-tuning. We argue that the standard L 2 regularization, which drives the parameters towards the origin, is not adequate in the framework of transfer learning where the initial values provide a more sensible reference point than the origin. This simple modification keeps the original control of overfitting, by constraining the effective search space around the initial solution, while encouraging committing to the acquired knowledge. We show that it has noticeable effects in inductive transfer learning scenarios.This paper aims at improving transfer learning by requiring less labeled training data. A form of transfer learning is thus considered, where some pieces of knowledge, acquired when solving a previous learning problem, have to be conveyed to another learning problem. Under this setting, we explore several parameter regularization methods that can explicitly retain the knowledge acquired on the source problem. We investigate variants of L 2 penalties using the pre-trained model as reference, which we name L 2 -SP because the pre-trained parameters represent the starting point (-SP) of the fine-tuning process. In addition, we evaluate other regularizers based on the Lasso and Group-Lasso penalties, which can freeze some individual parameters or groups of parameters to the pre-trained parameters. Fisher information is also taken into account when we test L 2 -SP and Group-Lasso-SP approaches.Our experiments indicate that all tested parameter regularization methods using the pre-trained parameters as a reference get an edge over the standard L 2 weight decay approach. We also analyze the effect of L 2 -SP with theoretical arguments and experimental evidence to recommend using L 2 -SP for transfer learning tasks. Among all -SP methods, L 2 -SP and L 2 -SP-Fisher always reach a better accuracy on the target task. We expected L 2 -SP-Fisher to outperform L 2 -SP, since Fisher information helps in continual learning, but there is no significant difference between the two options. Since L 2 -SP is simpler than L 2 -SP-Fisher, we recommend the former, and we focus on the analysis of L 2 -SP, although most of the analysis and the discussion would also apply to L 2 -SP-Fisher. We proposed simple regularization techniques for inductive transfer learning, to encode an explicit bias towards the solution learned on the source task. Most of the regularizers evaluated here have been already used for other purposes, but we demonstrate their relevance for inductive transfer learning with deep convolutional networks.We show that a simple L 2 penalty using the starting point as a reference, L 2 -SP, is useful, even if early stopping is used. This penalty is much more effective than the standard L 2 penalty that is commonly used in fine-tuning. It is also more effective and simpler to implement than the strategy consisting in freezing the first layers of a network. We provide theoretical hints and strong experimental evidence showing that L 2 -SP retains the memory of the features learned on the source database.Besides, we tested the effect of more elaborate penalties, based on L 1 or Group-L 1 norms, or based on Fisher information. None of the L 1 or Group-L 1 options seem to be valuable in the context of inductive transfer learning that we considered here, and using the Fisher information with L 2 -SP does not improve accuracy on the target task. Different approaches, which implement an implicit bias at the functional level, alike BID16 ), remain to be tested: being based on a different principle, their value should be assessed in the framework of inductive transfer learning.
In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. Addresses the problem of transfer learning in deep networks and proposes to have a regularization term that penalizes divergence from initialization. Proposes an analysis on different adaptive regularization techniques for deep transfer learning, specifically focusing on the use of an L@-SP condition