Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for future tasks to be added and thus leave enough capacity for them. To achieve Selfless Sequential Learning we study different regularization strategies and activation functions. We find that
 imposing sparsity at the level of the representation (i.e. neuron activations) is more beneficial for sequential learning than encouraging parameter sparsity. In particular, we propose a novel regularizer, that encourages representation sparsity by means of neural inhibition. It results in few active neurons which in turn leaves more free neurons to be utilized by upcoming tasks. As neural inhibition over an entire layer can be too drastic, especially for complex tasks requiring strong representations,
 our regularizer only inhibits other neurons in a local neighbourhood, inspired by lateral inhibition processes in the brain. We combine our novel regularizer with state-of-the-art lifelong learning methods that penalize changes to important previously learned parts of the network. We show that our new regularizer leads to increased sparsity which translates in consistent performance improvement on diverse datasets. Sequential learning, also referred to as continual, incremental, or lifelong learning (LLL), studies the problem of learning a sequence of tasks, one at a time, without access to the training data of previous or future tasks. When learning a new task, a key challenge in this context is how to avoid catastrophic interference with the tasks learned previously BID11 BID25 . Some methods exploit an additional episodic memory to store a small amount of previous tasks data to regularize future task learning (e.g. BID28 ). Others store previous tasks models and at test time, select one model or merge the models BID1 BID23 . In contrast, in this work we are interested in the challenging situation of learning a sequence of tasks without access to any previous or future task data and restricted to a fixed model capacity, as also studied in ; ; BID8 ; BID31 ; BID38 . This scenario not only has many practical benefits, including privacy and scalability, but also resembles more closely how the mammalian brain learns tasks over time.The mammalian brain is composed of billions of neurons. Yet at any given time, information is represented by only a few active neurons resulting in a sparsity of 90-95% BID24 . In neural biology, lateral inhibition describes the process where an activated neuron reduces the activity of its weaker neighbors. This creates a powerful decorrelated and compact representation with minimum interference between different input patterns in the brain BID49 . This is in stark contrast with artificial neural networks, which typically learn dense representations that are highly entangled BID3 . Such an entangled representation is quite sensitive to changes in the case. First layer indicates input patterns. Learning the first task utilizes parts indicated in red. Task 2 has different input patterns and uses parts shown in green. Orange indicates changed neurons activations as a result of the second task. In (a), when an example from the first task is encountered again, the activations of the first layer will not be affected by the changes, however, the second and later layer activations are changed. Such interference is largely reduced when imposing sparsity on the representation (b).input patterns, in that it responds differently to input patterns with only small variations. BID11 suggests that an overlapped internal representation plays a crucial role in catastrophic forgetting and reducing this overlap would result in a reduced interference. BID5 show that when the amount of overfitting in a neural network is reduced, the representation correlation is also reduced. As such , learning a disentangled representation is more powerful and less vulnerable to catastrophic interference. However , if the learned disentangled representation at a given task is not sparse, only little capacity is left for the learning of new tasks. This would in turn result in either an underfitting to the new tasks or again a forgetting of previous tasks. In contrast , a sparse and decorrelated representation would lead to a powerful representation and at the same time enough free neurons that can be changed without interference with the neural activations learned for the previous tasks.In general, sparsity in neural networks can be thought of either in terms of the network parameters or in terms of the representation (i.e., the activations). In this paper we postulate, and confirm experimentally, that a sparse and decorrelated representation is preferable over parameter sparsity in a sequential learning scenario. There are two arguments for this: first, a sparse representation is less sensitive to new and different patterns (such as data from new tasks) and second, the training procedure of the new tasks can use the free neurons leading to less interference with the previous tasks, hence reducing forgetting. In contrast, when the effective parameters are spread among different neurons, changing the ineffective ones would change the function of their corresponding neurons and hence interfere with previous tasks (see also FIG0 ). Based on these observations, we propose a new regularizer that exhibits a behavior similar to the lateral inhibition in biological neurons. The main idea of our regularizer is to penalize neurons that are active at the same time. This leads to more sparsity and a decorrelated representation. However, complex tasks may actually require multiple active neurons in a layer at the same time to learn a strong representation. Therefore, our regularizer , Sparse coding through Local Neural Inhibition and Discounting (SLNID), only penalizes neurons locally. Furthermore, we don't want inhibition to affect previously learned tasks, even if later tasks use neurons from earlier tasks. An important component of SLNID is thus to discount inhibition from/to neurons which have high neuron importance -a new concept that we introduce in analogy to parameter importance BID50 . When combined with a state-of-the-art important parameters preservation method , our proposed regularizer leads to sparse and decorrelated representations which improves the lifelong learning performance.Our contribution is threefold. First, we direct attention to Selfless Sequential Learning and study a diverse set of representation based regularizers, parameter based regularizers, as well as sparsity inducing activation functions to this end. These have not been studied extensively in the lifelong learning literature before. Second, we propose a novel regularizer, SLNID, which is inspired by lateral inhibition in the brain. Third, we show that our proposed regularizer consistently outperforms alternatives on three diverse datasets (Permuted MNIST, CIFAR, Tiny Imagenet) and we compare to and outperform state-of-the-art LLL approaches on an 8-task object classification challenge. SLNID can be applied to different regularization based LLL approaches, and we show experiments with MAS and EWC .In the following, we first discuss related approaches to LLL and different regularization criteria from a LLL perspective (Section 2). We proceed by introducing Selfless Sequential Learning and detailing our novel regularizer (Section 3). Section 4 describes our experimental evaluation, while Section 5 concludes the paper. In this paper we study the problem of sequential learning using a network with fixed capacity -a prerequisite for a scalable and computationally efficient solution. A key insight of our approach is that in the context of sequential learning (as opposed to other contexts where sparsity is imposed, such as network compression or avoiding overfitting), sparsity should be imposed at the level of the representation rather than at the level of the network parameters. Inspired by lateral inhibition in the mammalian brain, we impose sparsity by means of a new regularizer that decorrelates nearby active neurons. We integrate this in a model which learns selflessly a new task by leaving capacity for future tasks and at the same time avoids forgetting previous tasks by taking into account neurons importance.
A regularization strategy for improving the performance of sequential learning A novel, regularization based approach to the sequential learning problem using a fixed size model that adds extra terms to the loss, encouraging representation sparsity and combating catastrophic forgetting. This paper deals with the problem of catastrophic forgetting in lifelong learning by proposing regularized learning strategies