Sequential Learning in Neural Networks: A Review and a Discussion of Pseudorehearsal Based Methods
---
paper_title: Competitive Learning: From Interactive Activation to Adaptive Resonance

paper_content:

Functional ond mechanistic comparisons are mode between several network models of cognitive processing: competitive learning, interactive activation, adaptive resonance, and back propagation. The starting point of this comparison is the article of Rumelhart ond Zipser (1985) on feature discovery through competitive learning. All the models which Rumelhart and Zipser (1985) have described were shown in Grossberg (1976b) to exhibit a type of learning which is temporally unstable. Competitive learning mechanisms con be stabilized in response to an arbitrary input environment by being supplemented with mechanisms for learning top-down expectancies, or templates; for matching bottom-up input patterns with the top-down expectancies; and for releasing orienting reactions in o mismatch situation, thereby updating short-term memory ond searching for another internal representation. Network architectures which embody all of these mechonisms were called adaptive resonance models by Grossberg (1976~). Self-stabilizing learning models are candidates for use in real-world applications where unpredictable changes can occur in complex input environments. Competitive learning postulates ore inconsistent with the postulates of the interactive activotion model of McClelland and Rumelhart (1981). and suggest different levels of processing and interaction rules for the analysis of word recognition. Adaptive resonance models use these alternative levels and interaction rules. The selforganizing learning of on odaptive resonance model is compared ond contrasted with the teacher-directed learning of a back propagation model. A number of criteria for evaluating reol-time network models of cognitive processing ore described and applied.
---
paper_title: Learning and Categorization in Modular Neural Networks

paper_content:

Part 1 CALM - Categorizing And Learning Module: introduction description of CALM simulation studies of perfromance and self-organization in CALM. Part 2 Application: psychological models pattern recognition as a practical application genetic algorithms - modularity, learning and network design. Part 3 Evaluation of CALM.
---
paper_title: An analysis of catastrophic interference.

paper_content:

A number of recent simulation studies have shown that when feedforward neural nets are trained, using backpropagation, to memorize sets of items in sequential blocks and without negative exemplars, severe retroactive interference or catastrophic forgetting results. Both formal analysis and simulation studies are employed here to show why and under what circumstances such retroactive interference arises. The conclusion is that, on the one hand, approximations to 'ideal' network geometries can entirely alleviate interference if the training data sets have been generated from a learnable function (not arbitrary pattern associations). All that is required is either a representative training set or enough sequential memory sets. However, this elimination of interference comes with cost of a breakdown in discrimination between input patterns that have been learned and those that have not: catastrophic remembering. On the other hand, localized geometries for subfunctions eliminate the discrimination problem but ...
---
paper_title: Pseudopatterns and dual-network memory models: Advantages and shortcomings

paper_content:

The dual-network memory model is designed to be a neurobiologically plausible manner of avoiding catastrophic interference. We discuss a number of advantages of this model and potential clues that the model has provided in the areas of memory consolidation, category-specific deficits, anterograde and retrograde amnesia. We discuss a surprising result about how this class of models handles episodic (“snap-shot”) memory — namely, that they seem to be able to handle both episodic and abstract memory — and discuss two other promising areas of research involving these models.
---
paper_title: Control of consolidation in neural networks: avoiding runaway effects

paper_content:

Consolidation has been implemented in two ways: as straight rehearsal of patterns or as pseudorehearsal, in which pseudoitems are created by sampling attractors or input-output combinations from the network. Although both implementations have been investigated by several authors, few have explored how it is decided which pattern or pseudoitem is consolidated. Controlling consolidation is not trivial, as it is susceptible to a corruption. In runaway consolidation, one or two patterns monopolize all consolidation resources and come to dominate the entire network. Runaway consolidation is analysed, and three solutions are explored. Suppressing transmission in the connections in which consolidation takes place is shown to work best. Placing bounds on connections or unlearning attractors also alleviates runaway consolidation, though less effectively so.
---
paper_title: Neural networks with a self-refreshing memory: Knowledge transfer in sequential learning tasks without catastrophic forgetting

paper_content:

We explore a dual-network architecture with self-refreshing memory (Ans and Rousset 1997) which overcomes catastrophic forgetting in sequential learning tasks. Its principle is that new knowledge is learned along with an internally generated activity reflecting the network history. What mainly distinguishes this model from others using pseudorehearsal in feedforward multilayer networks is a reverberating process used for generating pseudoitems. This process, which tends to go up to network attractors from random activation, is more suitable for capturing optimally the deep structure of previously learned knowledge than a single feedforward pass of activity. The proposed mechanism for ?transporting memory? without loss of information between two different brain structures could be viewed as a neurobiologically plausible means for consolidation in long-term memory. Knowledge transfer is explored with regard to learning speed, ability to generalize and vulnerability to network damages. We show that transfer ...
---
paper_title: Dynamically Constraining Connectionist Networks to Produce Distributed, Orthogonal Representations to Reduce Catastrophic Interference

paper_content:

It is well known that when a connectionist network is trained on one set of patterns and then attempts to add new patterns to its repertoire, catastrophic interference may result. The use of sparse, orthogonal hidden-layer representations has been shown to reduce catastrophic interference. The author demonstrates that the use of sparse representations may, in certain cases, actually result in worse performance on catastrophic interference. This paper argues for the necessity of maintaining hidden-layer representations that are both as highly distributed and as highly orthogonal as possible. The author presents a learning algorithm, called context-biasing, that dynamically solves the problem of constraining hiddenlayer representations to simultaneously produce good orthogonality and distributedness. On the data tested for this study, context-biasing is shown to reduce catastrophic interference by more than 50% compared to standard backpropagation. In particular, this technique succeeds in reducing catastrophic interference on data where sparse, orthogonal distributions failed to produce any improvement.
---
paper_title: Catastrophic forgetting, rehearsal and pseudorehearsal

paper_content:

This paper reviews the problem of catastrophic forgetting (the loss or disruption of previously learned information when new information is learned) in neural networks, and explores rehearsal mechanisms (the retraining of some of the previously learned information as the new information is added) as a potential solution. We replicate some of the experiments described by Ratcliff (1990), including those relating to a simple 'recency' based rehearsal regime. We then develop further rehearsal regimes which are more effective than recency rehearsal. In particular, 'sweep rehearsal' is very successful at minimizing catastrophic forgetting. One possible limitation of rehearsal in general, however, is that previously learned information may not be available for retraining. We describe a solution to this problem, 'pseudorehearsal', a method which provides the advantages of rehearsal without actually requiring any access to the previously learned information (the original training population) itself. We then sugge...
---
paper_title: Consolidation in Neural Networks and in the Sleeping Brain

paper_content:

In this paper we explore the topic of the consolidation of information in neural network learning. One problem in particular has limited the ability of a broad range of neural networks to perform ongoing learning and consolidation. This is 'catastrophic forgetting', the tendency for new information, when it is learned, to disrupt old information. We will review and slightly extend the rehearsal and pseudorehearsal solutions to the catastrophic forgetting problem presented in Robins (1995). The main focus of this paper is to then relate these mechanisms to the consolidation processes which have been proposed in the psychological literature regarding sleep. We suggest that the catastrophic forgetting problem in artificial neural networks (ANNs) is a problem that has actually occurred in the evolution of the mammalian brain, and that the pseudorehearsal solution to the problem in ANNs is functionally equivalent to the sleep consolidation solution adopted by the brain. Finally, we review related work by McCle...
---
paper_title: Using Noise to Compute Error Surfaces in Connectionist Networks: A Novel Means of Reducing Catastrophic Forgetting

paper_content:

In error-driven distributed feedforward networks, new information typically interferes, sometimes severely, with previously learned information. We show how noise can be used to approximate the error surface of previously learned information. By combining this approximated error surface with the error surface associated with the new information to be learned, the network's retention of previously learned items can be improved and catastrophic interference significantly reduced. Further, we show that the noise-generated error surface is produced using only first-derivative information and without recourse to any explicit error information.
---
paper_title: Catastrophic forgetting and the pseudorehearsal solution in hopfield type networks

paper_content:

Pseudorehearsal is a mechanism proposed by Robins which alleviates catastrophic forgetting in multi-layer perceptron networks. In this paper, we extend the exploration of pseudorehearsal to a Hopfield-type net. The same general principles apply: old information can be rehearsed if it is available, and if it is not available, then generating and rehearsing approximations of old information that 'map' the behaviour of the network can also be effective at preserving the actual old information itself. The details of the pseudorehearsal mechanism, however, benefit from being adapted to the dynamics of Hopfield nets so as to exploit the extra attractors created in state space during learning. These attractors are usually described as 'spurious' or 'cross-talk', and regarded as undesirable, interfering with the retention of the trained population items. Our simulations have shown that, in another sense, such attractors can in fact be useful in preserving the learned population. In general terms, a solution to th...
---
paper_title: Local Learning Algorithms for Sequential Tasks in Neural Networks

paper_content:

In this paper we explore the concept of sequential learning and the efficacy of global and local neural network learning algorithms on a sequential learning task. Pseudorehearsal (a method developed by Robins [19] to solve the catastrophic forgetting problem which arises from the excessive plasticity of neural networks) is significantly more effective than other local learning algorithms for the sequential task. We further consider the concept of local learning and suggest that pseudorehearsal is so effective because it works directly at the level of the learned function, and not indirectly on the representation of the function within the network. We also briefly explore the effect of local learning on generalisation within the task.
---
paper_title: 10 – Catastrophic interference in neural networks: Causes, solutions, and data

paper_content:

Publisher Summary ::: Implausible forgetting characteristics are one of the main problems of neural networks. This chapter explores the causes underlying catastrophic interference and discusses two techniques to solve the problem. One technique is activation sharpening, which was applied to an existing connectionist model of cognitive deficits in schizophrenia. The results show that the revised model has reasonable forgetting characteristics and accounts for the impaired memory and accelerated forgetting observed in most schizophrenics. Interference manipulations necessarily present difficulties for the inherently ahistorical learning rule of connectionist nets. Techniques exist that can reduce interference without losing predictive power elsewhere. Appropriately modified networks can be successfully applied to forgetting paradigms, including those involving individual differences.
---
paper_title: Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions

paper_content:

Multilayer connectionist models of memory based on the encoder model using the backpropagation learning rule are evaluated. The models are applied to standard recognition memory procedures in which items are studied sequentially and then tested for retention. Sequential learning in these models leads to 2 major problems. First, well-learned information is forgotten rapidly as new information is learned. Second, discrimination between studied items and new items either decreases or is nonmonotonic as a function of learning. To address these problems, manipulations of the network within the multilayer model and several variants of the multilayer model were examined, including a model with prelearned memory and a context model, but none solved the problems. The problems discussed provide limitations on connectionist models applied to human memory and in tasks where information to be learned is not all available during learning. The first stage of the connectionist revolution in psychology is reaching maturity and perhaps drawing to an end. This stage has been concerned with the exploration of classes of models, and the criteria that have been used to evaluate the success of an application have been necessarily loose. In the early stages of development of a new approach, lax acceptability criteria are appropriate because of the large range of models to be examined. However, there comes a second stage when the models serve as competitors to existing models developed within other theoretical frameworks, and they have to be competitively evaluated according to more stringent criteria. A few notable connectionist models have reached these standards, whereas others have not. The second stage of development also requires that the connectionist models be evaluated in areas where their potential for success is not immediately obvious. One such area is recognition memory. The work presented in this article evaluates several variants of the multilayer connectionist model as accounts of empirical results in this area. I mainly discuss multilayer models using the error-correcting backpropagation algorithm and do not address other architectures such as adaptive resonance schemes (Carpenter & Grossberg, 1987). Before launching into the modeling of recognition memory, I need to specify the aims and rules under which this project was carried out. This is important in a new area of inquiry because there are many divergent views about what needs to be
---
paper_title: Catastrophic Forgetting in Connectionist Networks: Causes, Consequences and Solutions

paper_content:

All natural cognitive systems, and, in particular, our own, gradually forget previously learned information. Consequently, plausible models of human cognition should exhibit similar patterns of gradual forgetting old information as new information is acquired. Only rarely (see Box 3) does new learning in natural cognitive systems completely disrupt or erase previously learned information. In other words, natural cognitive systems do not, in general, forget catastrophically. Unfortunately, however, this is precisely what occurs under certain circumstances in distributed connectionist networks. It turns out that the very features that give these networks their much-touted abilities to generalize, to function in the presence of degraded input, etc., are the root cause of catastrophic forgetting. The challenge is how to keep the advantages of distributed connectionist networks while avoiding the problem of catastrophic forgetting. In this article, we examine the causes, consequences and numerous solutions to the problem of catastrophic forgetting in neural networks. We consider how the brain might have overcome this problem and explore the consequences of this solution.
---
paper_title: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory

paper_content:

Damage to the hippocampal system disrupts recent memory but leaves remote memory intact. Our account of this suggests that memories are first stored via synaptic changes in the hippocampal system; that these changes support reinstatement of recent memories in the neocortex; that neocortical synapses change a little on each reinstatement; and that remote memory is based on accumulated neocortical changes. Models that learn via adaptive changes to connections help explain this organization. These models discover the structure in ensembles of items if learning of each item is gradual and interleaved with learning about other items. This suggests that neocortex learns slowly to discover the structure in ensembles of experiences. The hippocampal system permits rapid learning of new items without disrupting this structure, and reinstatement of new memories interleaves them with others to integrate them into structured neocortical memory systems. Psychological Review, in press
---
paper_title: Selective transfer of neural network task knowledge

paper_content:

Within the context of artificial neural networks (ANN), we explore the question: How can a learning system retain and use previously learned knowledge to facilitate future learning? The research objectives are to develop a theoretical model and test a prototype system which sequentially retains ANN task knowledge and selectively uses that knowledge to bias the learning of a new task in an efficient and effective manner. ::: A theory of selective functional transfer is presented that requires a learning algorithm that employs a measure of task relatedness. ηMTL is introduced as a knowledge based inductive learning method that learns one or more secondary tasks within a back-propagation ANN as a source of inductive bias for a primary task. ηMTL employs a separate learning rate, ηk, for each secondary task output k. ηk varies as a function of a measure of relatedness, Rk, between the kth secondary task and the primary task of interest. Three categories of a priori measures of relatedness are developed for controlling inductive bias. ::: The task rehearsal method (TRM) is introduced to address the issue of sequential retention and generation of learned task knowledge. The representations of successfully learned tasks are stored within a domain knowledge repository. Virtual training examples generated from domain knowledge are rehearsed as secondary tasks in parallel with each new task using either standard multiple task learning (MTL) or ηMTL. ::: TRM using ηMTL is tested as a method of selective knowledge transfer and sequential learning on two synthetic domains and one medical diagnostic domain. Experiments show that the TRM provides an excellent method of retaining and generating accurate functional task knowledge. Hypotheses generated are compared statistically to single task learning and MTL hypotheses. We conclude that selective knowledge transfer with ηMTL develops more effective hypotheses but not necessarily with greater efficiency. The a priori measures of relatedness demonstrate significant value on certain domains of tasks but have difficulty scaling to large numbers of tasks. Several issues identified during the research indicate the importance of consolidating a representational form of domain knowledge.
---
paper_title: Self-refreshing memory in artificial neural networks : learning temporal sequences without catastrophic forgetting

paper_content:

While humans forget gradually, highly distributed connectionist networks forget catastrophically: newly learned information often completely erases previously learned information. This is not just implausible cognitively, but disastrous practically. However, it is not easy in connectionist cognitive modelling to keep away from highly distributed neural networks, if only because of their ability to generalize. A realistic and effective system that solves the problem of catastrophic interference in sequential learning of ‘static’ (i.e. non-temporally ordered) patterns has been proposed recently (Robins 1995, Connection Science, 7: 123–146, 1996, Connection Science, 8: 259–275, Ans and Rousset 1997, CR Academie des Sciences Paris, Life Sciences, 320: 989–997, French 1997, Connection Science, 9: 353–379, 1999, Trends in Cognitive Sciences, 3: 128–135, Ans and Rousset 2000, Connection Science, 12: 1–19). The basic principle is to learn new external patterns interleaved with internally generated ‘pseudopatterns...
---
paper_title: Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks

paper_content:

Everyone agrees that real cognition requires much more than static pattern recognition. In particular, it requires the ability to learn sequences of patterns (or actions) But learning sequences really means being able to learn multiple sequences, one after the other, wi thout the most recently learned ones erasing the previously learned ones. But if catastrophic interference is a problem for the sequential learning of individual patterns, the problem is amplified many times over when multiple sequences of patterns have to be learned consecutively, because each new sequence consists of many linked patterns. In this paper we will present a connectionist architecture that would seem to solve the problem of multiple sequence learning using pseudopatterns.
---
paper_title: Semi-distributed representations and catastrophic forgetting in connectionist networks

paper_content:

Abstract A major problem with connectionist networks is that newly-learned information may completely destroy previously-learned information unless the network is continually retrained on the old information. This phenomenon, known as catastrophic forgetting, is unacceptable both for practical purposes and as a model of mind. This paper advances the claim that catastrophic forgetting is in part the result of the overlap of system's distributed representations and can be reduced by reducing this overlap. A simple algorithm, called activation sharpening, is presented that allows a standard feed-forward backpropagation network to develop semi-distributed representations, thereby reducing the problem of catastrophic forgetting. Activation sharpening is discussed in tight of recent work done by other researchers who have experimented with this and other techniques for reducing catastrophic forgetting.
---
paper_title: Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem

paper_content:

Publisher Summary Connectionist networks in which information is stored in weights on connections among simple processing units have attracted considerable interest in cognitive science. Much of the interest centers around two characteristics of these networks. First, the weights on connections between units need not be prewired by the model builder but rather may be established through training in which items to be learned are presented repeatedly to the network and the connection weights are adjusted in small increments according to a learning algorithm. Second, the networks may represent information in a distributed fashion. This chapter discusses the catastrophic interference in connectionist networks. Distributed representations established through the application of learning algorithms have several properties that are claimed to be desirable from the standpoint of modeling human cognition. These properties include content-addressable memory and so-called automatic generalization in which a network trained on a set of items responds correctly to other untrained items within the same domain. New learning may interfere catastrophically with old learning when networks are trained sequentially. The analysis of the causes of interference implies that at least some interference will occur whenever new learning may alter weights involved in representing old learning, and the simulation results demonstrate only that interference is catastrophic in some specific networks.
---
paper_title: An analysis of catastrophic interference.

paper_content:

A number of recent simulation studies have shown that when feedforward neural nets are trained, using backpropagation, to memorize sets of items in sequential blocks and without negative exemplars, severe retroactive interference or catastrophic forgetting results. Both formal analysis and simulation studies are employed here to show why and under what circumstances such retroactive interference arises. The conclusion is that, on the one hand, approximations to 'ideal' network geometries can entirely alleviate interference if the training data sets have been generated from a learnable function (not arbitrary pattern associations). All that is required is either a representative training set or enough sequential memory sets. However, this elimination of interference comes with cost of a breakdown in discrimination between input patterns that have been learned and those that have not: catastrophic remembering. On the other hand, localized geometries for subfunctions eliminate the discrimination problem but ...
---
paper_title: Catastrophic forgetting, rehearsal and pseudorehearsal

paper_content:

This paper reviews the problem of catastrophic forgetting (the loss or disruption of previously learned information when new information is learned) in neural networks, and explores rehearsal mechanisms (the retraining of some of the previously learned information as the new information is added) as a potential solution. We replicate some of the experiments described by Ratcliff (1990), including those relating to a simple 'recency' based rehearsal regime. We then develop further rehearsal regimes which are more effective than recency rehearsal. In particular, 'sweep rehearsal' is very successful at minimizing catastrophic forgetting. One possible limitation of rehearsal in general, however, is that previously learned information may not be available for retraining. We describe a solution to this problem, 'pseudorehearsal', a method which provides the advantages of rehearsal without actually requiring any access to the previously learned information (the original training population) itself. We then sugge...
---
paper_title: Consolidation in Neural Networks and in the Sleeping Brain

paper_content:

In this paper we explore the topic of the consolidation of information in neural network learning. One problem in particular has limited the ability of a broad range of neural networks to perform ongoing learning and consolidation. This is 'catastrophic forgetting', the tendency for new information, when it is learned, to disrupt old information. We will review and slightly extend the rehearsal and pseudorehearsal solutions to the catastrophic forgetting problem presented in Robins (1995). The main focus of this paper is to then relate these mechanisms to the consolidation processes which have been proposed in the psychological literature regarding sleep. We suggest that the catastrophic forgetting problem in artificial neural networks (ANNs) is a problem that has actually occurred in the evolution of the mammalian brain, and that the pseudorehearsal solution to the problem in ANNs is functionally equivalent to the sleep consolidation solution adopted by the brain. Finally, we review related work by McCle...
---
paper_title: Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions

paper_content:

Multilayer connectionist models of memory based on the encoder model using the backpropagation learning rule are evaluated. The models are applied to standard recognition memory procedures in which items are studied sequentially and then tested for retention. Sequential learning in these models leads to 2 major problems. First, well-learned information is forgotten rapidly as new information is learned. Second, discrimination between studied items and new items either decreases or is nonmonotonic as a function of learning. To address these problems, manipulations of the network within the multilayer model and several variants of the multilayer model were examined, including a model with prelearned memory and a context model, but none solved the problems. The problems discussed provide limitations on connectionist models applied to human memory and in tasks where information to be learned is not all available during learning. The first stage of the connectionist revolution in psychology is reaching maturity and perhaps drawing to an end. This stage has been concerned with the exploration of classes of models, and the criteria that have been used to evaluate the success of an application have been necessarily loose. In the early stages of development of a new approach, lax acceptability criteria are appropriate because of the large range of models to be examined. However, there comes a second stage when the models serve as competitors to existing models developed within other theoretical frameworks, and they have to be competitively evaluated according to more stringent criteria. A few notable connectionist models have reached these standards, whereas others have not. The second stage of development also requires that the connectionist models be evaluated in areas where their potential for success is not immediately obvious. One such area is recognition memory. The work presented in this article evaluates several variants of the multilayer connectionist model as accounts of empirical results in this area. I mainly discuss multilayer models using the error-correcting backpropagation algorithm and do not address other architectures such as adaptive resonance schemes (Carpenter & Grossberg, 1987). Before launching into the modeling of recognition memory, I need to specify the aims and rules under which this project was carried out. This is important in a new area of inquiry because there are many divergent views about what needs to be
---
paper_title: Catastrophic Forgetting in Connectionist Networks: Causes, Consequences and Solutions

paper_content:

All natural cognitive systems, and, in particular, our own, gradually forget previously learned information. Consequently, plausible models of human cognition should exhibit similar patterns of gradual forgetting old information as new information is acquired. Only rarely (see Box 3) does new learning in natural cognitive systems completely disrupt or erase previously learned information. In other words, natural cognitive systems do not, in general, forget catastrophically. Unfortunately, however, this is precisely what occurs under certain circumstances in distributed connectionist networks. It turns out that the very features that give these networks their much-touted abilities to generalize, to function in the presence of degraded input, etc., are the root cause of catastrophic forgetting. The challenge is how to keep the advantages of distributed connectionist networks while avoiding the problem of catastrophic forgetting. In this article, we examine the causes, consequences and numerous solutions to the problem of catastrophic forgetting in neural networks. We consider how the brain might have overcome this problem and explore the consequences of this solution.
---
paper_title: Semi-distributed representations and catastrophic forgetting in connectionist networks

paper_content:

Abstract A major problem with connectionist networks is that newly-learned information may completely destroy previously-learned information unless the network is continually retrained on the old information. This phenomenon, known as catastrophic forgetting, is unacceptable both for practical purposes and as a model of mind. This paper advances the claim that catastrophic forgetting is in part the result of the overlap of system's distributed representations and can be reduced by reducing this overlap. A simple algorithm, called activation sharpening, is presented that allows a standard feed-forward backpropagation network to develop semi-distributed representations, thereby reducing the problem of catastrophic forgetting. Activation sharpening is discussed in tight of recent work done by other researchers who have experimented with this and other techniques for reducing catastrophic forgetting.
---
paper_title: Understanding catastrophic interference in neural nets

paper_content:

An electro-optic device comprises a plurality of first electrodes provided on a top substrate and a second electrode provided on a bottom substrate, the first electrodes confronting the second electrode through electro-optical matter which will show electro-optical phenomena by the application of electric field, the first and second electrodes being connected through external impedance elements such as resistors or capacitors to a variable voltage source. In this electro-optic device, the electric field applied between the first and second electrodes are partially changed by various means, for instance, by making the areas of the first electrodes different, or by varying the values of the external impedance elements such as resistors or capacitors, as a result of which the electro-optical matter shows various electro-optical phenomena.
---
paper_title: Learning and Categorization in Modular Neural Networks

paper_content:

Part 1 CALM - Categorizing And Learning Module: introduction description of CALM simulation studies of perfromance and self-organization in CALM. Part 2 Application: psychological models pattern recognition as a practical application genetic algorithms - modularity, learning and network design. Part 3 Evaluation of CALM.
---
paper_title: The Cascade-Correlation Learning Architecture

paper_content:

Cascade-Correlation is a new architecture and supervised learning algorithm for artificial neural networks. Instead of just adjusting the weights in a network of fixed topology. Cascade-Correlation begins with a minimal network, then automatically trains and adds new hidden units one by one, creating a multi-layer structure. Once a new hidden unit has been added to the network, its input-side weights are frozen. This unit then becomes a permanent feature-detector in the network, available for producing outputs or for creating other, more complex feature detectors. The Cascade-Correlation architecture has several advantages over existing algorithms: it learns very quickly, the network determines its own size and topology, it retains the structures it has built even if the training set changes, and it requires no back-propagation of error signals through the connections of the network.
---
paper_title: Neural networks with a self-refreshing memory: Knowledge transfer in sequential learning tasks without catastrophic forgetting

paper_content:

We explore a dual-network architecture with self-refreshing memory (Ans and Rousset 1997) which overcomes catastrophic forgetting in sequential learning tasks. Its principle is that new knowledge is learned along with an internally generated activity reflecting the network history. What mainly distinguishes this model from others using pseudorehearsal in feedforward multilayer networks is a reverberating process used for generating pseudoitems. This process, which tends to go up to network attractors from random activation, is more suitable for capturing optimally the deep structure of previously learned knowledge than a single feedforward pass of activity. The proposed mechanism for ?transporting memory? without loss of information between two different brain structures could be viewed as a neurobiologically plausible means for consolidation in long-term memory. Knowledge transfer is explored with regard to learning speed, ability to generalize and vulnerability to network damages. We show that transfer ...
---
paper_title: ALCOVE : An exemplar-based connectionist model of category learning

paper_content:

ALCOVE (attention learning covering map) is a connectionist model of category learning that incorporates an exemplar-based representation (Medin & Schaffer, 1978; Nosofsky, 1986) with error-driven learning (Gluck & Bower, 1988; Rumelhart, Hinton, & Williams, 1986). ALCOVE selectively attends to relevant stimilus dimensions, is sensitive to correlated dimensions, can account for a form of base-rate neglect, does not suffer catastrophic forgetting, and can exhibit 3-stage (U-shaped) learning of high-frequency exceptions to rules, whereas such effects are not easily accounted for by models using other combinations of representation and learning method
---
paper_title: Connectionist Models and Their Properties

paper_content:

Much of the progress in the fields constituting cognitive science has been based upon the use of explicit information processing models, almost exclusively patterned after conventional serial computers. An extension of these ideas to massively parallel, connectionist models appears to offer a number of advantages. After a preliminary discussion, this paper introduces a general connectionist model and considers how it might be used in cognitive science. Among the issues addressed are: stability and noise-sensitivity, distributed decision-making, time and sequence problems, and the representation of complex concepts.
---
paper_title: Catastrophic Forgetting in Connectionist Networks: Causes, Consequences and Solutions

paper_content:

All natural cognitive systems, and, in particular, our own, gradually forget previously learned information. Consequently, plausible models of human cognition should exhibit similar patterns of gradual forgetting old information as new information is acquired. Only rarely (see Box 3) does new learning in natural cognitive systems completely disrupt or erase previously learned information. In other words, natural cognitive systems do not, in general, forget catastrophically. Unfortunately, however, this is precisely what occurs under certain circumstances in distributed connectionist networks. It turns out that the very features that give these networks their much-touted abilities to generalize, to function in the presence of degraded input, etc., are the root cause of catastrophic forgetting. The challenge is how to keep the advantages of distributed connectionist networks while avoiding the problem of catastrophic forgetting. In this article, we examine the causes, consequences and numerous solutions to the problem of catastrophic forgetting in neural networks. We consider how the brain might have overcome this problem and explore the consequences of this solution.
---
paper_title: The ART of adaptive pattern recognition by a self-organizing neural network

paper_content:

The adaptive resonance theory (ART) suggests a solution to the stability-plasticity dilemma facing designers of learning systems, namely how to design a learning system that will remain plastic, or adaptive, in response to significant events and yet remain stable in response to irrelevant events. ART architectures are discussed that are neural networks that self-organize stable recognition codes in real time in response to arbitrary sequences of input patterns. Within such an ART architecture, the process of adaptive pattern recognition is a special case of the more general cognitive process of hypothesis discovery, testing, search, classification, and learning. This property opens up the possibility of applying ART systems to more general problems of adaptively processing large abstract information sources and databases. The main computational properties of these ART architectures are outlined and contrasted with those of alternative learning and recognition systems. >
---
paper_title: Dynamically Constraining Connectionist Networks to Produce Distributed, Orthogonal Representations to Reduce Catastrophic Interference

paper_content:

It is well known that when a connectionist network is trained on one set of patterns and then attempts to add new patterns to its repertoire, catastrophic interference may result. The use of sparse, orthogonal hidden-layer representations has been shown to reduce catastrophic interference. The author demonstrates that the use of sparse representations may, in certain cases, actually result in worse performance on catastrophic interference. This paper argues for the necessity of maintaining hidden-layer representations that are both as highly distributed and as highly orthogonal as possible. The author presents a learning algorithm, called context-biasing, that dynamically solves the problem of constraining hiddenlayer representations to simultaneously produce good orthogonality and distributedness. On the data tested for this study, context-biasing is shown to reduce catastrophic interference by more than 50% compared to standard backpropagation. In particular, this technique succeeds in reducing catastrophic interference on data where sparse, orthogonal distributions failed to produce any improvement.
---
paper_title: Catastrophic Forgetting in Connectionist Networks: Causes, Consequences and Solutions

paper_content:

All natural cognitive systems, and, in particular, our own, gradually forget previously learned information. Consequently, plausible models of human cognition should exhibit similar patterns of gradual forgetting old information as new information is acquired. Only rarely (see Box 3) does new learning in natural cognitive systems completely disrupt or erase previously learned information. In other words, natural cognitive systems do not, in general, forget catastrophically. Unfortunately, however, this is precisely what occurs under certain circumstances in distributed connectionist networks. It turns out that the very features that give these networks their much-touted abilities to generalize, to function in the presence of degraded input, etc., are the root cause of catastrophic forgetting. The challenge is how to keep the advantages of distributed connectionist networks while avoiding the problem of catastrophic forgetting. In this article, we examine the causes, consequences and numerous solutions to the problem of catastrophic forgetting in neural networks. We consider how the brain might have overcome this problem and explore the consequences of this solution.
---
paper_title: Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks

paper_content:

Everyone agrees that real cognition requires much more than static pattern recognition. In particular, it requires the ability to learn sequences of patterns (or actions) But learning sequences really means being able to learn multiple sequences, one after the other, wi thout the most recently learned ones erasing the previously learned ones. But if catastrophic interference is a problem for the sequential learning of individual patterns, the problem is amplified many times over when multiple sequences of patterns have to be learned consecutively, because each new sequence consists of many linked patterns. In this paper we will present a connectionist architecture that would seem to solve the problem of multiple sequence learning using pseudopatterns.
---
paper_title: Semi-distributed representations and catastrophic forgetting in connectionist networks

paper_content:

Abstract A major problem with connectionist networks is that newly-learned information may completely destroy previously-learned information unless the network is continually retrained on the old information. This phenomenon, known as catastrophic forgetting, is unacceptable both for practical purposes and as a model of mind. This paper advances the claim that catastrophic forgetting is in part the result of the overlap of system's distributed representations and can be reduced by reducing this overlap. A simple algorithm, called activation sharpening, is presented that allows a standard feed-forward backpropagation network to develop semi-distributed representations, thereby reducing the problem of catastrophic forgetting. Activation sharpening is discussed in tight of recent work done by other researchers who have experimented with this and other techniques for reducing catastrophic forgetting.
---
paper_title: Neural networks with a self-refreshing memory: Knowledge transfer in sequential learning tasks without catastrophic forgetting

paper_content:

We explore a dual-network architecture with self-refreshing memory (Ans and Rousset 1997) which overcomes catastrophic forgetting in sequential learning tasks. Its principle is that new knowledge is learned along with an internally generated activity reflecting the network history. What mainly distinguishes this model from others using pseudorehearsal in feedforward multilayer networks is a reverberating process used for generating pseudoitems. This process, which tends to go up to network attractors from random activation, is more suitable for capturing optimally the deep structure of previously learned knowledge than a single feedforward pass of activity. The proposed mechanism for ?transporting memory? without loss of information between two different brain structures could be viewed as a neurobiologically plausible means for consolidation in long-term memory. Knowledge transfer is explored with regard to learning speed, ability to generalize and vulnerability to network damages. We show that transfer ...
---
paper_title: Catastrophic forgetting, rehearsal and pseudorehearsal

paper_content:

This paper reviews the problem of catastrophic forgetting (the loss or disruption of previously learned information when new information is learned) in neural networks, and explores rehearsal mechanisms (the retraining of some of the previously learned information as the new information is added) as a potential solution. We replicate some of the experiments described by Ratcliff (1990), including those relating to a simple 'recency' based rehearsal regime. We then develop further rehearsal regimes which are more effective than recency rehearsal. In particular, 'sweep rehearsal' is very successful at minimizing catastrophic forgetting. One possible limitation of rehearsal in general, however, is that previously learned information may not be available for retraining. We describe a solution to this problem, 'pseudorehearsal', a method which provides the advantages of rehearsal without actually requiring any access to the previously learned information (the original training population) itself. We then sugge...
---
paper_title: Consolidation in Neural Networks and in the Sleeping Brain

paper_content:

In this paper we explore the topic of the consolidation of information in neural network learning. One problem in particular has limited the ability of a broad range of neural networks to perform ongoing learning and consolidation. This is 'catastrophic forgetting', the tendency for new information, when it is learned, to disrupt old information. We will review and slightly extend the rehearsal and pseudorehearsal solutions to the catastrophic forgetting problem presented in Robins (1995). The main focus of this paper is to then relate these mechanisms to the consolidation processes which have been proposed in the psychological literature regarding sleep. We suggest that the catastrophic forgetting problem in artificial neural networks (ANNs) is a problem that has actually occurred in the evolution of the mammalian brain, and that the pseudorehearsal solution to the problem in ANNs is functionally equivalent to the sleep consolidation solution adopted by the brain. Finally, we review related work by McCle...
---
paper_title: Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions

paper_content:

Multilayer connectionist models of memory based on the encoder model using the backpropagation learning rule are evaluated. The models are applied to standard recognition memory procedures in which items are studied sequentially and then tested for retention. Sequential learning in these models leads to 2 major problems. First, well-learned information is forgotten rapidly as new information is learned. Second, discrimination between studied items and new items either decreases or is nonmonotonic as a function of learning. To address these problems, manipulations of the network within the multilayer model and several variants of the multilayer model were examined, including a model with prelearned memory and a context model, but none solved the problems. The problems discussed provide limitations on connectionist models applied to human memory and in tasks where information to be learned is not all available during learning. The first stage of the connectionist revolution in psychology is reaching maturity and perhaps drawing to an end. This stage has been concerned with the exploration of classes of models, and the criteria that have been used to evaluate the success of an application have been necessarily loose. In the early stages of development of a new approach, lax acceptability criteria are appropriate because of the large range of models to be examined. However, there comes a second stage when the models serve as competitors to existing models developed within other theoretical frameworks, and they have to be competitively evaluated according to more stringent criteria. A few notable connectionist models have reached these standards, whereas others have not. The second stage of development also requires that the connectionist models be evaluated in areas where their potential for success is not immediately obvious. One such area is recognition memory. The work presented in this article evaluates several variants of the multilayer connectionist model as accounts of empirical results in this area. I mainly discuss multilayer models using the error-correcting backpropagation algorithm and do not address other architectures such as adaptive resonance schemes (Carpenter & Grossberg, 1987). Before launching into the modeling of recognition memory, I need to specify the aims and rules under which this project was carried out. This is important in a new area of inquiry because there are many divergent views about what needs to be
---
paper_title: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory

paper_content:

Damage to the hippocampal system disrupts recent memory but leaves remote memory intact. Our account of this suggests that memories are first stored via synaptic changes in the hippocampal system; that these changes support reinstatement of recent memories in the neocortex; that neocortical synapses change a little on each reinstatement; and that remote memory is based on accumulated neocortical changes. Models that learn via adaptive changes to connections help explain this organization. These models discover the structure in ensembles of items if learning of each item is gradual and interleaved with learning about other items. This suggests that neocortex learns slowly to discover the structure in ensembles of experiences. The hippocampal system permits rapid learning of new items without disrupting this structure, and reinstatement of new memories interleaves them with others to integrate them into structured neocortical memory systems. Psychological Review, in press
---
paper_title: Selective transfer of neural network task knowledge

paper_content:

Within the context of artificial neural networks (ANN), we explore the question: How can a learning system retain and use previously learned knowledge to facilitate future learning? The research objectives are to develop a theoretical model and test a prototype system which sequentially retains ANN task knowledge and selectively uses that knowledge to bias the learning of a new task in an efficient and effective manner. ::: A theory of selective functional transfer is presented that requires a learning algorithm that employs a measure of task relatedness. ηMTL is introduced as a knowledge based inductive learning method that learns one or more secondary tasks within a back-propagation ANN as a source of inductive bias for a primary task. ηMTL employs a separate learning rate, ηk, for each secondary task output k. ηk varies as a function of a measure of relatedness, Rk, between the kth secondary task and the primary task of interest. Three categories of a priori measures of relatedness are developed for controlling inductive bias. ::: The task rehearsal method (TRM) is introduced to address the issue of sequential retention and generation of learned task knowledge. The representations of successfully learned tasks are stored within a domain knowledge repository. Virtual training examples generated from domain knowledge are rehearsed as secondary tasks in parallel with each new task using either standard multiple task learning (MTL) or ηMTL. ::: TRM using ηMTL is tested as a method of selective knowledge transfer and sequential learning on two synthetic domains and one medical diagnostic domain. Experiments show that the TRM provides an excellent method of retaining and generating accurate functional task knowledge. Hypotheses generated are compared statistically to single task learning and MTL hypotheses. We conclude that selective knowledge transfer with ηMTL develops more effective hypotheses but not necessarily with greater efficiency. The a priori measures of relatedness demonstrate significant value on certain domains of tasks but have difficulty scaling to large numbers of tasks. Several issues identified during the research indicate the importance of consolidating a representational form of domain knowledge.
---
paper_title: Self-refreshing memory in artificial neural networks : learning temporal sequences without catastrophic forgetting

paper_content:

While humans forget gradually, highly distributed connectionist networks forget catastrophically: newly learned information often completely erases previously learned information. This is not just implausible cognitively, but disastrous practically. However, it is not easy in connectionist cognitive modelling to keep away from highly distributed neural networks, if only because of their ability to generalize. A realistic and effective system that solves the problem of catastrophic interference in sequential learning of ‘static’ (i.e. non-temporally ordered) patterns has been proposed recently (Robins 1995, Connection Science, 7: 123–146, 1996, Connection Science, 8: 259–275, Ans and Rousset 1997, CR Academie des Sciences Paris, Life Sciences, 320: 989–997, French 1997, Connection Science, 9: 353–379, 1999, Trends in Cognitive Sciences, 3: 128–135, Ans and Rousset 2000, Connection Science, 12: 1–19). The basic principle is to learn new external patterns interleaved with internally generated ‘pseudopatterns...
---
paper_title: Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks

paper_content:

Everyone agrees that real cognition requires much more than static pattern recognition. In particular, it requires the ability to learn sequences of patterns (or actions) But learning sequences really means being able to learn multiple sequences, one after the other, wi thout the most recently learned ones erasing the previously learned ones. But if catastrophic interference is a problem for the sequential learning of individual patterns, the problem is amplified many times over when multiple sequences of patterns have to be learned consecutively, because each new sequence consists of many linked patterns. In this paper we will present a connectionist architecture that would seem to solve the problem of multiple sequence learning using pseudopatterns.
---
paper_title: Fast Learning in Networks of Locally-Tuned Processing Units

paper_content:

We propose a network architecture which uses a single internal layer of locally-tuned processing units to learn both classification tasks and real-valued function approximations (Moody and Darken 1988). We consider training such networks in a completely supervised manner, but abandon this approach in favor of a more computationally efficient hybrid learning method which combines self-organized and supervised learning. Our networks learn faster than backpropagation for two reasons: the local representations ensure that only a few units respond to any given input, thus reducing computational overhead, and the hybrid learning rules are linear rather than nonlinear, thus leading to faster convergence. Unlike many existing methods for data analysis, our network architecture and learning rules are truly adaptive and are thus appropriate for real-time use.
---
paper_title: Pseudopatterns and dual-network memory models: Advantages and shortcomings

paper_content:

The dual-network memory model is designed to be a neurobiologically plausible manner of avoiding catastrophic interference. We discuss a number of advantages of this model and potential clues that the model has provided in the areas of memory consolidation, category-specific deficits, anterograde and retrograde amnesia. We discuss a surprising result about how this class of models handles episodic (“snap-shot”) memory — namely, that they seem to be able to handle both episodic and abstract memory — and discuss two other promising areas of research involving these models.
---
paper_title: Pattern Classification And Scene Analysis

paper_content:

Provides a unified, comprehensive and up-to-date treatment of both statistical and descriptive methods for pattern recognition. The topics treated include Bayesian decision theory, supervised and unsupervised learning, nonparametric techniques, discriminant analysis, clustering, preprosessing of pictorial data, spatial filtering, shape description techniques, perspective transformations, projective invariants, linguistic procedures, and artificial intelligence techniques for scene analysis.
---
paper_title: Local Learning Algorithms for Sequential Tasks in Neural Networks

paper_content:

In this paper we explore the concept of sequential learning and the efficacy of global and local neural network learning algorithms on a sequential learning task. Pseudorehearsal (a method developed by Robins [19] to solve the catastrophic forgetting problem which arises from the excessive plasticity of neural networks) is significantly more effective than other local learning algorithms for the sequential task. We further consider the concept of local learning and suggest that pseudorehearsal is so effective because it works directly at the level of the learned function, and not indirectly on the representation of the function within the network. We also briefly explore the effect of local learning on generalisation within the task.
---
paper_title: Semi-distributed representations and catastrophic forgetting in connectionist networks

paper_content:

Abstract A major problem with connectionist networks is that newly-learned information may completely destroy previously-learned information unless the network is continually retrained on the old information. This phenomenon, known as catastrophic forgetting, is unacceptable both for practical purposes and as a model of mind. This paper advances the claim that catastrophic forgetting is in part the result of the overlap of system's distributed representations and can be reduced by reducing this overlap. A simple algorithm, called activation sharpening, is presented that allows a standard feed-forward backpropagation network to develop semi-distributed representations, thereby reducing the problem of catastrophic forgetting. Activation sharpening is discussed in tight of recent work done by other researchers who have experimented with this and other techniques for reducing catastrophic forgetting.
---
paper_title: Catastrophic forgetting, rehearsal and pseudorehearsal

paper_content:

This paper reviews the problem of catastrophic forgetting (the loss or disruption of previously learned information when new information is learned) in neural networks, and explores rehearsal mechanisms (the retraining of some of the previously learned information as the new information is added) as a potential solution. We replicate some of the experiments described by Ratcliff (1990), including those relating to a simple 'recency' based rehearsal regime. We then develop further rehearsal regimes which are more effective than recency rehearsal. In particular, 'sweep rehearsal' is very successful at minimizing catastrophic forgetting. One possible limitation of rehearsal in general, however, is that previously learned information may not be available for retraining. We describe a solution to this problem, 'pseudorehearsal', a method which provides the advantages of rehearsal without actually requiring any access to the previously learned information (the original training population) itself. We then sugge...
---
paper_title: Pseudopatterns and dual-network memory models: Advantages and shortcomings

paper_content:

The dual-network memory model is designed to be a neurobiologically plausible manner of avoiding catastrophic interference. We discuss a number of advantages of this model and potential clues that the model has provided in the areas of memory consolidation, category-specific deficits, anterograde and retrograde amnesia. We discuss a surprising result about how this class of models handles episodic (“snap-shot”) memory — namely, that they seem to be able to handle both episodic and abstract memory — and discuss two other promising areas of research involving these models.
---
paper_title: The list-strength effect: II. Theoretical mechanisms

paper_content:

Ratcliff, Clark, and Shiffrin (1990) examined the list-strength effect: the effect of strengthening (or weakening) some list items upon memory for other list items. The list-strength effect was missing or negative in recognition, missing or positive in cued recall, and large and positive in free recall
---
paper_title: Using Noise to Compute Error Surfaces in Connectionist Networks: A Novel Means of Reducing Catastrophic Forgetting

paper_content:

In error-driven distributed feedforward networks, new information typically interferes, sometimes severely, with previously learned information. We show how noise can be used to approximate the error surface of previously learned information. By combining this approximated error surface with the error surface associated with the new information to be learned, the network's retention of previously learned items can be improved and catastrophic interference significantly reduced. Further, we show that the noise-generated error surface is produced using only first-derivative information and without recourse to any explicit error information.
---
paper_title: Retrieval Processes in Recognition Memory

paper_content:

A method of analyzing reaction time data in recognition memory is presented, which uses an explicit model of latency distributions. This distributional method allows us to distinguish between processes in a way that the traditional measure, mean latency, can not. The behavior of latency distributions is described, and four experiments are reported that show how recognition accuracy and latency vary with independent variables such as study and test position, rate of presentation, and list length. These data are used to develop and test the empirical model. The resulting analyses, together with functional relationships derived from the experimental data, are used to test several theories of recognition memory. The theories examined all show problems in light of these stringent tests, and general properties required by a model to account for the data are suggested. As well as arguing for distributional analyses of reaction time data, this paper presents a wide range of phenomena that any theory of recognition memory must explain. Over the last few years, researchers have been developing theories of recognition memory based not only on accuracy measures but also on latency measures. In this article, we consider latency measures in recognition memory. Results from four experiments are presented, and an empirical model for latency distributions is developed. Latency distributions are shown to provide much more information than can be obtained from mean latency, the most common dependent variable in reaction time measurements. From this, a strong case is made for the study of distributional properties by showing how some current theories are inadequate or wrong when examined in the light of distributiona l analyses. These recent theories are further evaluated using functional relationships extracted from results of the four experiments presented.
---
paper_title: Self-refreshing memory in artificial neural networks : learning temporal sequences without catastrophic forgetting

paper_content:

While humans forget gradually, highly distributed connectionist networks forget catastrophically: newly learned information often completely erases previously learned information. This is not just implausible cognitively, but disastrous practically. However, it is not easy in connectionist cognitive modelling to keep away from highly distributed neural networks, if only because of their ability to generalize. A realistic and effective system that solves the problem of catastrophic interference in sequential learning of ‘static’ (i.e. non-temporally ordered) patterns has been proposed recently (Robins 1995, Connection Science, 7: 123–146, 1996, Connection Science, 8: 259–275, Ans and Rousset 1997, CR Academie des Sciences Paris, Life Sciences, 320: 989–997, French 1997, Connection Science, 9: 353–379, 1999, Trends in Cognitive Sciences, 3: 128–135, Ans and Rousset 2000, Connection Science, 12: 1–19). The basic principle is to learn new external patterns interleaved with internally generated ‘pseudopatterns...
---
paper_title: Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks

paper_content:

Everyone agrees that real cognition requires much more than static pattern recognition. In particular, it requires the ability to learn sequences of patterns (or actions) But learning sequences really means being able to learn multiple sequences, one after the other, wi thout the most recently learned ones erasing the previously learned ones. But if catastrophic interference is a problem for the sequential learning of individual patterns, the problem is amplified many times over when multiple sequences of patterns have to be learned consecutively, because each new sequence consists of many linked patterns. In this paper we will present a connectionist architecture that would seem to solve the problem of multiple sequence learning using pseudopatterns.
---
paper_title: Hebbian learning, its correlation catastrophe, and unlearning

paper_content:

Learning and associative memory are concerned with storing and retrieving activity patterns of a neuronal net. It is considered to be a minimal requirement that the number of patterns that can be stored faithfully is extensive, i.e. is at least proportional to the number of neighbours each neuron interacts with. The main drawback of Hebbian learning, and of any one-shot local learning procedure, is that it cannot store extensively many patterns with activities which vary from pattern to pattern because, being local, it cannot discern global correlations. We critically review the performance of Hebbian unlearning—also proposed as a model of REM sleep—in a network of formal neurons with a distribution of axonal delays. Hebbian unlearning, though as local and unsupervised as Hebbian learning, eliminates undesirable global correlations, handles any spatio-temporal pattern, and improves the network performance greatly—sometimes even saturating a theoretical upper bound. Furthermore, it is shown that unlearning...
---
paper_title: Neural networks and physical systems with emergent collective computational abilities

paper_content:

Abstract ::: Computational properties of use of biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.
---
paper_title: Investigation of the Crick-Mitchison reverse-learning dream sleep hypothesis in a dynamical setting

paper_content:

Abstract Crick and Mitchison have proposed that dream sleep involves a reverse-learning (or unlearning) mechanism by which the brain eliminates internal parasitic modes from memory and avoids the overloading catastrophe. We perform stimulations on the Hopfield neural network to test this hypothesis. In our dynamical model, the neural network is allowed to learn (while awake) and unlearn (while dreaming) in alternate cycles, in much the same was as in a biological system. Our results show that with the appropriate mix of learning and unlearning (dreaming) the neural network can continue to function for ever and ever, avoiding the overloading catastrophe. As found in other palimpsest models, the dynamic storage capacity in our model is lower in comparison to the static model. This would seem to suggest that the function of dream sleep may not be primarily concerned with improving the storage capacity of the brain as is commonly thought. On the basis of this we suggest that dream sleep may be more important as a means to gradually forget weak (that is, old, irrelevant and parasitic) memories.
---
paper_title: The ART of adaptive pattern recognition by a self-organizing neural network

paper_content:

The adaptive resonance theory (ART) suggests a solution to the stability-plasticity dilemma facing designers of learning systems, namely how to design a learning system that will remain plastic, or adaptive, in response to significant events and yet remain stable in response to irrelevant events. ART architectures are discussed that are neural networks that self-organize stable recognition codes in real time in response to arbitrary sequences of input patterns. Within such an ART architecture, the process of adaptive pattern recognition is a special case of the more general cognitive process of hypothesis discovery, testing, search, classification, and learning. This property opens up the possibility of applying ART systems to more general problems of adaptively processing large abstract information sources and databases. The main computational properties of these ART architectures are outlined and contrasted with those of alternative learning and recognition systems. >
---
paper_title: The Cascade-Correlation Learning Architecture

paper_content:

Cascade-Correlation is a new architecture and supervised learning algorithm for artificial neural networks. Instead of just adjusting the weights in a network of fixed topology. Cascade-Correlation begins with a minimal network, then automatically trains and adds new hidden units one by one, creating a multi-layer structure. Once a new hidden unit has been added to the network, its input-side weights are frozen. This unit then becomes a permanent feature-detector in the network, available for producing outputs or for creating other, more complex feature detectors. The Cascade-Correlation architecture has several advantages over existing algorithms: it learns very quickly, the network determines its own size and topology, it retains the structures it has built even if the training set changes, and it requires no back-propagation of error signals through the connections of the network.
---
paper_title: Maximum Storage Capacity in Neural Networks

paper_content:

The upper storage capacity of a neural network for patterns of fixed magnetization m is calculated. The optimal capacity increases with the correlation m2 between the patterns.
---
paper_title: Control of consolidation in neural networks: avoiding runaway effects

paper_content:

Consolidation has been implemented in two ways: as straight rehearsal of patterns or as pseudorehearsal, in which pseudoitems are created by sampling attractors or input-output combinations from the network. Although both implementations have been investigated by several authors, few have explored how it is decided which pattern or pseudoitem is consolidated. Controlling consolidation is not trivial, as it is susceptible to a corruption. In runaway consolidation, one or two patterns monopolize all consolidation resources and come to dominate the entire network. Runaway consolidation is analysed, and three solutions are explored. Suppressing transmission in the connections in which consolidation takes place is shown to work best. Placing bounds on connections or unlearning attractors also alleviates runaway consolidation, though less effectively so.
---
paper_title: Training with noise and the storage of correlated patterns in a neural network model

paper_content:

Local iterative learning algorithms for the interactions between Ising spins in neural network models are discussed. They converge to solutions with basins of attraction whose shape is determined by the noise in the training data, provided such solutions exist. The training is applied both to the storage of random patterns and to a model for the storage of correlated words. The existence of correlations increases the storage capacity of a given network beyond that for random patterns. The model can be modified to store cycles of patterns and in particular is applied to the storage of continuous items of English text.
---
paper_title: Pattern Classification And Scene Analysis

paper_content:

Provides a unified, comprehensive and up-to-date treatment of both statistical and descriptive methods for pattern recognition. The topics treated include Bayesian decision theory, supervised and unsupervised learning, nonparametric techniques, discriminant analysis, clustering, preprosessing of pictorial data, spatial filtering, shape description techniques, perspective transformations, projective invariants, linguistic procedures, and artificial intelligence techniques for scene analysis.
---
paper_title: Catastrophic forgetting and the pseudorehearsal solution in hopfield type networks

paper_content:

Pseudorehearsal is a mechanism proposed by Robins which alleviates catastrophic forgetting in multi-layer perceptron networks. In this paper, we extend the exploration of pseudorehearsal to a Hopfield-type net. The same general principles apply: old information can be rehearsed if it is available, and if it is not available, then generating and rehearsing approximations of old information that 'map' the behaviour of the network can also be effective at preserving the actual old information itself. The details of the pseudorehearsal mechanism, however, benefit from being adapted to the dynamics of Hopfield nets so as to exploit the extra attractors created in state space during learning. These attractors are usually described as 'spurious' or 'cross-talk', and regarded as undesirable, interfering with the retention of the trained population items. Our simulations have shown that, in another sense, such attractors can in fact be useful in preserving the learned population. In general terms, a solution to th...
---
paper_title: Investigation of the Crick-Mitchison reverse-learning dream sleep hypothesis in a dynamical setting

paper_content:

Abstract Crick and Mitchison have proposed that dream sleep involves a reverse-learning (or unlearning) mechanism by which the brain eliminates internal parasitic modes from memory and avoids the overloading catastrophe. We perform stimulations on the Hopfield neural network to test this hypothesis. In our dynamical model, the neural network is allowed to learn (while awake) and unlearn (while dreaming) in alternate cycles, in much the same was as in a biological system. Our results show that with the appropriate mix of learning and unlearning (dreaming) the neural network can continue to function for ever and ever, avoiding the overloading catastrophe. As found in other palimpsest models, the dynamic storage capacity in our model is lower in comparison to the static model. This would seem to suggest that the function of dream sleep may not be primarily concerned with improving the storage capacity of the brain as is commonly thought. On the basis of this we suggest that dream sleep may be more important as a means to gradually forget weak (that is, old, irrelevant and parasitic) memories.
---
paper_title: A "Thermal" Perceptron Learning Rule

paper_content:

The thermal perceptron is a simple extension to Rosenblatt’s perceptron learning rule for training individual linear threshold units. It finds stable weights for nonseparable problems as well as separable ones. Experiments indicate that if a good initial setting for a temperature parameter, To, has been found, then the thermal perceptron outperforms the Pocket algorithm and methods based on gradient descent. The learning rule stabilizes the weights (learns) over a fixed training period. For separable problems it finds separating weights much more quickly than the usual rules.
---
paper_title: A robust method for distinguishing between learned and spurious attractors

paper_content:

Hopfield/constraint satisfaction type networks can be used to learn (autoassociate) patterns. Random inputs to the network will sometimes converge on states which are learned patterns, and sometimes converge on states which are unlearned/spurious. It would be useful for many reasons to be able to tell whether or not a given state was learned or spurious. In this paper we present a robust and general method, based on 'energy profiles', which allows us to make this distinction. We briefly describe related research, and note links with the study of recall, recognition and familiarity in the psychological literature.
---
paper_title: Catastrophic forgetting, rehearsal and pseudorehearsal

paper_content:

This paper reviews the problem of catastrophic forgetting (the loss or disruption of previously learned information when new information is learned) in neural networks, and explores rehearsal mechanisms (the retraining of some of the previously learned information as the new information is added) as a potential solution. We replicate some of the experiments described by Ratcliff (1990), including those relating to a simple 'recency' based rehearsal regime. We then develop further rehearsal regimes which are more effective than recency rehearsal. In particular, 'sweep rehearsal' is very successful at minimizing catastrophic forgetting. One possible limitation of rehearsal in general, however, is that previously learned information may not be available for retraining. We describe a solution to this problem, 'pseudorehearsal', a method which provides the advantages of rehearsal without actually requiring any access to the previously learned information (the original training population) itself. We then sugge...
---
paper_title: Consolidation in Neural Networks and in the Sleeping Brain

paper_content:

In this paper we explore the topic of the consolidation of information in neural network learning. One problem in particular has limited the ability of a broad range of neural networks to perform ongoing learning and consolidation. This is 'catastrophic forgetting', the tendency for new information, when it is learned, to disrupt old information. We will review and slightly extend the rehearsal and pseudorehearsal solutions to the catastrophic forgetting problem presented in Robins (1995). The main focus of this paper is to then relate these mechanisms to the consolidation processes which have been proposed in the psychological literature regarding sleep. We suggest that the catastrophic forgetting problem in artificial neural networks (ANNs) is a problem that has actually occurred in the evolution of the mammalian brain, and that the pseudorehearsal solution to the problem in ANNs is functionally equivalent to the sleep consolidation solution adopted by the brain. Finally, we review related work by McCle...
---
paper_title: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory

paper_content:

Damage to the hippocampal system disrupts recent memory but leaves remote memory intact. Our account of this suggests that memories are first stored via synaptic changes in the hippocampal system; that these changes support reinstatement of recent memories in the neocortex; that neocortical synapses change a little on each reinstatement; and that remote memory is based on accumulated neocortical changes. Models that learn via adaptive changes to connections help explain this organization. These models discover the structure in ensembles of items if learning of each item is gradual and interleaved with learning about other items. This suggests that neocortex learns slowly to discover the structure in ensembles of experiences. The hippocampal system permits rapid learning of new items without disrupting this structure, and reinstatement of new memories interleaves them with others to integrate them into structured neocortical memory systems. Psychological Review, in press
---

<format>
Title: Sequential Learning in Neural Networks: A Review and a Discussion of Pseudorehearsal Based Methods
Section 1: Introduction
Description 1: Introduction to the problem of sequential learning and catastrophic forgetting in neural networks, and the purpose of the paper.

Section 2: Catastrophic forgetting
Description 2: Overview of the CF problem and its implications for neural networks, with a focus on back propagation type networks.

Section 3: The problem
Description 3: Detailed explanation of how CF occurs in neural networks with an illustrative example.

Section 4: Scope of the problem
Description 4: Discussion of the severity and scope of CF in various types of neural networks.

Section 5: Reducing overlap
Description 5: Description of methods proposed to counteract CF by reducing the overlap in neural network representations.

Section 6: The pseudorehearsal solution
Description 6: Introduction to pseudorehearsal as an approach to mitigate CF.

Section 7: Rehearsal and pseudorehearsal
Description 7: In-depth discussion of rehearsal and pseudorehearsal methods in the context of back propagation networks.

Section 8: Functions and local learning
Description 8: Explanation of why pseudorehearsal works through the lens of neural networks functioning as approximators and local learning methods.

Section 9: Storage issues
Description 9: Exploration of storage challenges in pseudorehearsal and potential solutions for generating pseudoitems on-demand.

Section 10: Pseudorehearsal based ANNs
Description 10: Examination of various neural network architectures implementing pseudorehearsal without an external pseudoitem store.

Section 11: Dual networks
Description 11: Description of the dual network method, an approach for implementing pseudorehearsal within neural networks.

Section 12: Dual weights
Description 12: Overview of a dual weight approach for integrating pseudorehearsal within a single neural network.

Section 13: Other architectures
Description 13: Discussion of other potential architectural approaches to implementing pseudorehearsal in neural networks.

Section 14: Further results
Description 14: Summary of additional results and phenomena arising from the study of pseudorehearsal techniques.

Section 15: Hopfield type networks
Description 15: Exploration of CF and pseudorehearsal in the context of Hopfield type neural networks.

Section 16: Capacity and forgetting  
Description 16: Discussion of capacity-related issues and forgetting in Hopfield networks, and comparison with unlearning methods.

Section 17: Forgetting with error correcting learning 
Description 17: Examination of how CF and forgetting are handled by Hopfield type networks that use error-correcting learning algorithms like the delta rule.

Section 18: Discussion
Description 18: Comparative analysis of different methods to mitigate CF, their cognitive implications, and the relevance to biological neural systems.

Section 19: Summary and discussion
Description 19: A comprehensive summary of the findings and discussions from previous sections, with concluding remarks on CF solutions.
</format>
