Incremental class learning involves sequentially learning classes in bursts of examples from the same class. This violates the assumptions that underlie  methods for training standard deep neural networks, and will cause them to suffer from catastrophic forgetting. Arguably, the best method for incremental class learning is iCaRL, but it requires storing  training examples for each class, making it challenging to scale. Here, we propose FearNet for incremental class learning. FearNet is a generative model that does not store previous examples, making it memory efficient. FearNet uses a brain-inspired dual-memory system in which new memories are consolidated from a network for recent memories inspired by the mammalian hippocampal complex to a network for long-term storage inspired by medial prefrontal cortex. Memory consolidation is inspired by mechanisms that occur during sleep. FearNet also uses a module inspired by the basolateral amygdala for determining which memory system to use for recall.   FearNet achieves state-of-the-art performance at incremental class learning on image (CIFAR-100, CUB-200) and audio classification (AudioSet) benchmarks.
 In incremental classification, an agent must sequentially learn to classify training examples, without necessarily having the ability to re-study previously seen examples. While deep neural networks (DNNs) have revolutionized machine perception BID26 , off-the-shelf DNNs cannot incrementally learn classes due to catastrophic forgetting. Catastrophic forgetting is a phenomenon in which a DNN completely fails to learn new data without forgetting much of its previously learned knowledge BID29 . While methods have been developed to try and mitigate catastrophic forgetting, as shown in BID23 , these methods are not sufficient and perform poorly on larger datasets. In this paper, we propose FearNet, a brain-inspired system for incrementally learning categories that significantly outperforms previous methods.The standard way for dealing with catastrophic forgetting in DNNs is to avoid it altogether by mixing new training examples with old ones and completely re-training the model offline. For large datasets, this may require weeks of time, and it is not a scalable solution. An ideal incremental learning system would be able to assimilate new information without the need to store the entire training dataset. A major application for incremental learning includes real-time operation on-board embedded platforms that have limited computing power, storage, and memory, e.g., smart toys, smartphone applications, and robots. For example, a toy robot may need to learn to recognize objects within its local environment and of interest to its owner. Using cloud computing to overcome these resource limitations may pose privacy risks and may not be scalable to a large number of embedded devices. A better solution is on-device incremental learning, which requires the model to use less storage and computational power.In this paper, we propose an incremental learning framework called FearNet (see Fig. 1 ). FearNet has three brain-inspired sub-systems: 1) a recent memory system for quick recall, 2) a memory system for long-term storage, and 3) a sub-system that determines which memory system to use for a particular example. FearNet mitigates catastrophic forgetting by consolidating recent memories into long-term storage using pseudorehearsal BID34 . Pseudorehearsal allows the network to revisit previous memories during incremental training without the need to store previous training examples, which is more memory efficient.Figure 1: FearNet consists of three braininspired modules based on 1) mPFC (longterm storage), 2) HC (recent storage), and 3) BLA for determining whether to use mPFC or HC for recall.Problem Formulation: Here, incremental class learning consists of T study-sessions. At time t, the learner receives a batch of data B t , which contains N t labeled training samples, i.e., B t = {(x j , y j )} Nt j=1 , where x j ∈ R d is the input feature vector to be classified and y j is its corresponding label. The number of training samples N t may vary between sessions, and the data inside a study-session is not assumed to be independent and identically distributed (iid). During a study session, the learner only has access to its current batch, but it may use its own memory to store information from prior study sessions. We refer to the first session as the model's "base-knowledge," which contains exemplars from M ≥ 1 classes. The batches learned in all subsequent sessions contain only one class, i.e., all y j will be identical within those sessions.Novel Contributions: Our contributions include:1. FearNet's architecture includes three neural networks: one inspired by the hippocampal complex (HC) for recent memories, one inspired by the medial prefrontal cortex (mPFC) for long-term storage, and one inspired by the basolateral amygdala (BLA) that determines whether to use HC or mPFC for recall.2. Motivated by memory replay during sleep, FearNet employs a generative autoencoder for pseudorehearsal, which mitigates catastrophic forgetting by generating previously learned examples that are replayed alongside novel information during consolidation. This process does not involve storing previous training data.3. FearNet achieves state-of-the-art results on large image and audio datasets with a relatively small memory footprint, demonstrating how dual-memory models can be scaled. FearNet's mPFC is trained to both discriminate examples and also generate new examples. While the main use of mPFC's generative abilities is to enable psuedorehearsal, this ability may also help make the model more robust to catastrophic forgetting. BID18 observed that unsupervised networks are more robust (but not immune) to catastrophic forgetting because there are no target outputs to be forgotten. Since the pseudoexample generator is learned as a unsupervised reconstruction task, this could explain why FearNet is slow to forget old information. Table 5 : Memory requirements to train CIFAR-100 and the amount of memory that would be required if these models were trained up to 1,000 classes. Table 5 shows the memory requirements for each model in Sec. 6.1 for learning CIFAR-100 and a hypothetical extrapolation for learning 1,000 classes. This chart accounts for a fixed model capacity and storage of any data or class statistics. FearNet's memory footprint is comparatively small because it only stores class statistics rather than some or all of the raw training data, which makes it better suited for deployment.An open question is how to deal with storage and updating of class statistics if classes are seen in more than one study sessions. One possibility is to use a running update for the class means and covariances, but it may be better to favor the data from the most recent study session due to learning in the autoencoder.FearNet assumed that the output of the mPFC encoder was normally distributed for each class, which may not be the case. It would be interesting to consider modeling the classes with a more complex model, e.g., a Gaussian Mixture Model. BID34 showed that pseudorehearsal worked reasonably well with randomly generated vectors because they were associated with the weights of a given class. Replaying these vectors strengthened their corresponding weights, which could be what is happening with the pseudo-examples generated by FearNet's decoder. The largest impact on model size is the stored covariance matrix Σ c for each class. We tested a variant of FearNet that used a diagonal Σ c instead of a full covariance matrix. TAB5 shows that performance degrades, but FearNet still works.FearNet can be adapted to other paradigms, such as unsupervised learning and regression. For unsupervised learning, FearNet's mPFC already does a form of it implicitly. For regression, this would require changing mPFC's loss function and may require grouping input feature vectors into similar collections. FearNet could also be adapted to perform the supervised data permutation experiment performed by BID20 and BID24 . This would likely require storing statistics from previous permutations and classes. FearNet would sleep between learning different permutations; however, if the number of classes was high, recent recall may suffer. In this paper, we proposed a brain-inspired framework capable of incrementally learning data with different modalities and object classes. FearNet outperforms existing methods for incremental class learning on large image and audio classification benchmarks, demonstrating that FearNet is capable of recalling and consolidating recently learned information while also retaining old information. In addition, we showed that FearNet is more memory efficient, making it ideal for platforms where size, weight, and power requirements are limited. Future work will include 1) integrating BLA directly into the model (versus training it independently); 2) replacing HC with a semi-parametric model; 3) learning the feature embedding from raw inputs; and 4) replacing the pseduorehearsal mechanism with a generative model that does not require the storage of class statistics, which would be more memory efficient.A SUPPLEMENTAL MATERIAL A.1 MODEL HYPERPARAMETERS TAB1 shows the training parameters for the FearNet model for each dataset. We also experimented with various dropout rates, weight decay, and various activation functions; however, weight decay did not work well with FearNet's mPFC. TAB1 : FearNet Training Parameters TAB2 shows the training parameters for the iCaRL framework used in this paper. We adapted the code from the author's GitHub page for our own experiments. The ResNet-18 convolutional neural network was replaced with a fully-connected neural network. We experimented with various regularization strategies to increase the initial base-knowledge accuracy with weight decay working the best. The values that are given as a range of values are the hyperparameter search spaces. TAB9 shows the training parameters for GeppNet and GeppNet+STM. Parameters not listed here are the default parameters defined by BID17 . The values that are given as a range of values are the hyperparameter search spaces. A.3 BLA VARIANTS Our BLA model is a classifier that determines whether a prediction should be made using HC (recent memory) or mPFC (remote memory). An alternative approach would be to use an outlier detection algorithm that determines whether the data being processed by a sub-network is an outlier for that sub-network and should therefore be processed by the other sub-network. To explore this alternative BLA formulation, we experimented with three outlier detection algorithms: 1) one-class support vector machine (SVM) BID36 , 2) determining if the data fits into a Gaussian distribution using a minimum covariance determinant estimation (i.e., elliptical envelope) (Rousseeuw BID35 , and 3) the isolation forest BID27 . All three of these methods set a rejection criterion for if the test sample exists in HC; whereas the binary MLP reports a probability on how likely the test sample resides in HC. TAB5 : Performance of different BLA variants.
FearNet is a memory efficient neural-network, inspired by memory formation in the mammalian brain, that is capable of incremental class learning without catastrophic forgetting. This paper presents a novel solution to an incremental classification problem based on a dual memory system. 