MODULAR NEURAL NETWORKS: A SURVEY
---
paper_title: Application of neural networks to sorting problems

paper_content:

The authors present a design philosophy for constraint optimization problems which is based on viewing the neural network as an interconnection of several subsystems. Specifically, they present a method to classify ranges in the magnitude of an analog input signal and associate an arbitrary digital output with each range. The proposed approach is illustrated by application to a cascadable 2-bit analog/digital converter. The advantages of the proposed design method as compared with energy function designs are enumerated. >
---
paper_title: Comparison of brain structure to a backpropagation-learned-structure

paper_content:

This paper describes the results of experiments studying the circumstances under which an error-minimizing artificial neural network mimics the mammal visual system. The networks were trained to recognize handwritten-digits. The experiment was not intended to yield a high identification accuracy, but rather to generate a comparison of the neural networks to biology under different circumstances. Experiments were conducted with partially hand-set networks, freely-trained networks, and convolutionally constrained networks. The convolutional experiment, where certain weights were constrained to be identical, performed the best at digit recognition while also modeling parts of biology that we had not anticipated the network would model. Rather than using the input image to generate an edge detection outline, as occurs in the retina, the network's first layer modeled the cones themselves, reacting most to one color (black or white), but not performing any real processing.
---
paper_title: Learning trajectories with a hierarchy of oscillatory modules

paper_content:

The most successful approach to learning has been the backpropagation method. Although very powerful on relatively simple problems, theoretical analysis and simulations show that this approach breaks down as soon as sufficiently complex problems are considered. To overcome this fundamental limitation, a hierarchical and modular approach is suggested, directly inspired from biological networks, whereby a certain degree of structure is introduced in the learning system. This approach is applied to a simple example of trajectory learning of a semi-figure eight. >
---
paper_title: Comparison of brain structure to a backpropagation-learned-structure

paper_content:

This paper describes the results of experiments studying the circumstances under which an error-minimizing artificial neural network mimics the mammal visual system. The networks were trained to recognize handwritten-digits. The experiment was not intended to yield a high identification accuracy, but rather to generate a comparison of the neural networks to biology under different circumstances. Experiments were conducted with partially hand-set networks, freely-trained networks, and convolutionally constrained networks. The convolutional experiment, where certain weights were constrained to be identical, performed the best at digit recognition while also modeling parts of biology that we had not anticipated the network would model. Rather than using the input image to generate an edge detection outline, as occurs in the retina, the network's first layer modeled the cones themselves, reacting most to one color (black or white), but not performing any real processing.
---
paper_title: Capacity and Information Efficiency of a Brain-like Associative Net

paper_content:

We have determined the capacity and information efficiency of an associative net configured in a brain-like way with partial connectivity and noisy input cues. Recall theory was used to calculate the capacity when pattern recall is achieved using a winners-take-all strategy. Transforming the dendritic sum according to input activity and unit usage can greatly increase the capacity of the associative net under these conditions. For moderately sparse patterns, maximum information efficiency is achieved with very low connectivity levels (≤ 10%). This corresponds to the level of connectivity commonly seen in the brain and invites speculation that the brain is connected in the most information efficient way.
---
paper_title: A Multi-Scale Network Model of Brightness Perception

paper_content:

Air Force Office of Scientific Research (F49620-92-J-0334); Northeast Consortium for Engineering Education (NCEE-A303-21-93); Office of Naval Research (N00014-91-J-4100); German BMFT grant (413-5839-01 IN 101 C/1); CNPq and NUTES/UFRJ, Brazil
---
paper_title: Why are What and Where Processed by Separate Cortical Visual Systems? A Computational Investigation

paper_content:

In the primate visual system, the identification of objects and the processing of spatial information are accomplished by different cortical pathways. The computational properties of this “two-systems” design were explored by constructing simplifying connectionist models. The models were designed to simultaneously classify and locate shapes that could appear in multiple positions in a matrix, and the ease of forming representations of the two kinds of information was measured. Some networks were designed so that all hidden nodes projected to all output nodes, whereas others had the hidden nodes split into two groups, with some projecting to the output nodes that registered shape identity and the remainder projecting to the output nodes that registered location. The simulations revealed that splitting processing into separate streams for identifying and locating a shape led to better performance only under some circumstances. Provided that enough computational resources were available in both streams, split networks were able to develop more efficient internal representations, as revealed by detailed analyses of the patterns of weights between connections.
---
paper_title: Modular Learning in Neural Networks: A Modularized Approach to Neural Network Classification

paper_content:

From the Publisher: ::: This important new work recognizes the advanced nature of today's artificial neural networks, uniquely emphasizing a modular approach to neural network learning. By breaking down the learning task into relatively independent parts of lower complexity, Modular Learning in Neural Networks demonstrates how neural network learning can be made more powerful and efficient. The book's modular approach, unlike the monolithic viewpoint, admits intermediary solution stages whose success can be independently verified, as in other engineering fields. Each stage can be evaluated before moving on to the subsequent one, and the reason for possible failures can be analyzed, ultimately leading to the improved development and engineering of applications. The modular approach also takes into account growing network complexity, reducing the difficulty of such inevitable problems as scaling and convergence. Modular Learning in Neural Networks' modular approach is also fully in step with important psychological and neurobiological research. Studies in developmental psychology demonstrate the incremental nature of human learning, in which the success of each stage is conditioned by the successful accomplishment of the previous stage, while neurobiology has depicted the human brain as a complex structure of cooperating modules. Modular Learning in Neural Networks covers the full range of conceivable approaches to the modularization of learning, including decomposition of learning into modules using supervised and unsupervised learning types; decomposition of the function to be mapped into linear and nonlinear parts; decomposition of the neural network to minimize harmful interferences between a large number of network parameters during learning; decomposition of the application task into subtasks that are learned separately; decomposition into a knowledge-based part and a learning part. The book attempts to show that modular learning based on these approaches is helpful in improving t
---
paper_title: Hierarchical visual stimuli: electrophysiological evidence for separate left hemispheric global and local processing mechanisms in humans

paper_content:

Abstract Using reaction time measurements it has been shown that the neural processing of hierarchically composed visual stimuli may be mediated by partly distinctive mechanisms of global and local feature analysis. Patients with focal left hemisphere brain lesions showed an impairment of local and patients with right hemisphere lesions of global stimulus perception. Our experiment was designed to extend the investigations of global/local feature analysis using the event-related brain potential (ERP) technique in healthy subjects. Thirteen subjects viewed a series of hierarchically composed non-linguistic stimuli in a divided attention condition in order to detect target stimuli which could be present at both local and global feature level. ERP analysis showed a posterior negative component (Ne) to be the correlate of early global/local information processing. When local targets were presented there was an effect of the global (non-target) level (termed ‘distractor level’) upon the onset of Ne but not vice versa. This supports the view of separate cortical mechanisms of global/local feature analysis. At odds with the previous patient data was the finding that both global and local level information was predominantly processed from the left hemisphere.
---
paper_title: The cortical column: a new processing unit for multilayered networks

paper_content:

Abstract We propose in this paper a new connectionist unit that matches a biological model of the cortical column. The architectural and functional characteristics of this unit have been designed in the simplest manner in order to simulate human-like reasoning, and to be as similar as possible to the main known features of real intracortical networks. We use a new type of learning rule which can easily take into account goal-oriented combinations of actions in behavioral programs. These learning rules are both simple and biologically plausible. We show in this paper that such units can be used in multilayered networks to perform pattern recognition, with feedback connections effecting an attentive gating of sensory information flow. Computer simulations were performed to assess the ability of a multilayered network made of these biologically inspired units to perform standard speech and visual recognition. Such simulations show levels of performance equivalent to the best currently available connectionist networks for typical human-like problems, with very fast learning and recognition processes. Furthermore, this type of “cortical” unit can be used in more general multilayered networks with units controlling different types of external processing, in order to learn programs of actions which may be included in the process of recognition.
---
paper_title: The functional logic of cortical connections

paper_content:

Patterns of anatomical connections in the visual cortex form the structural basis for segregating features of the visual image into separate cortical areas and for communication between these areas at all levels to produce a coherent percept. Such multi-stage integration may be a common strategy throughout the cortex for producing complex behaviour.
---
paper_title: Modular Learning in Neural Networks: A Modularized Approach to Neural Network Classification

paper_content:

From the Publisher: ::: This important new work recognizes the advanced nature of today's artificial neural networks, uniquely emphasizing a modular approach to neural network learning. By breaking down the learning task into relatively independent parts of lower complexity, Modular Learning in Neural Networks demonstrates how neural network learning can be made more powerful and efficient. The book's modular approach, unlike the monolithic viewpoint, admits intermediary solution stages whose success can be independently verified, as in other engineering fields. Each stage can be evaluated before moving on to the subsequent one, and the reason for possible failures can be analyzed, ultimately leading to the improved development and engineering of applications. The modular approach also takes into account growing network complexity, reducing the difficulty of such inevitable problems as scaling and convergence. Modular Learning in Neural Networks' modular approach is also fully in step with important psychological and neurobiological research. Studies in developmental psychology demonstrate the incremental nature of human learning, in which the success of each stage is conditioned by the successful accomplishment of the previous stage, while neurobiology has depicted the human brain as a complex structure of cooperating modules. Modular Learning in Neural Networks covers the full range of conceivable approaches to the modularization of learning, including decomposition of learning into modules using supervised and unsupervised learning types; decomposition of the function to be mapped into linear and nonlinear parts; decomposition of the neural network to minimize harmful interferences between a large number of network parameters during learning; decomposition of the application task into subtasks that are learned separately; decomposition into a knowledge-based part and a learning part. The book attempts to show that modular learning based on these approaches is helpful in improving t
---
paper_title: Hierarchical visual stimuli: electrophysiological evidence for separate left hemispheric global and local processing mechanisms in humans

paper_content:

Abstract Using reaction time measurements it has been shown that the neural processing of hierarchically composed visual stimuli may be mediated by partly distinctive mechanisms of global and local feature analysis. Patients with focal left hemisphere brain lesions showed an impairment of local and patients with right hemisphere lesions of global stimulus perception. Our experiment was designed to extend the investigations of global/local feature analysis using the event-related brain potential (ERP) technique in healthy subjects. Thirteen subjects viewed a series of hierarchically composed non-linguistic stimuli in a divided attention condition in order to detect target stimuli which could be present at both local and global feature level. ERP analysis showed a posterior negative component (Ne) to be the correlate of early global/local information processing. When local targets were presented there was an effect of the global (non-target) level (termed ‘distractor level’) upon the onset of Ne but not vice versa. This supports the view of separate cortical mechanisms of global/local feature analysis. At odds with the previous patient data was the finding that both global and local level information was predominantly processed from the left hemisphere.
---
paper_title: Thumb-pointing in humans after damage to somatic sensory cortex

paper_content:

Three patients with a severe somatosensory deficit consequent on damage of the right somatosensory cortices were required, while blindfolded, to point with their insensate thumb to select positions on the other left fingers. Given the absence of feedback, the motor performance of the insensate thumb appeared grossly impaired in all patients. However, all patients attained end-points with an accuracy greater than chance. This result suggests that spatial accuracy may not rely entirely on sensory feedback. A good accuracy of pointing was evinced also in potentially facilitating conditions where somatosensory and motor cues coming from the intact side during simultaneous movement of both thumbs, vision of stimulated point and final thumb position, and visuomotor imagery were available. Furthermore, in one patient, the accuracy of the insensate thumb in cued conditions was higher than in a reference baseline condition, thus indicating that motor and cognitive cues can help the motor performance of patients with cortical somatosensory lesions.
---
paper_title: The functional logic of cortical connections

paper_content:

Patterns of anatomical connections in the visual cortex form the structural basis for segregating features of the visual image into separate cortical areas and for communication between these areas at all levels to produce a coherent percept. Such multi-stage integration may be a common strategy throughout the cortex for producing complex behaviour.
---
paper_title: An Adaptive Modular Neural Network With Application to Unconstrained Character Recognition

paper_content:

The topology and the capacity of a traditional multilayer neural system, as measured by the number of connections in the network, has surprisingly little impact on its generalization ability. This paper presents a new adaptive modular network that offers superior generalization capability. The new network provides significant fault tolerance, quick adaption to novel inputs, and high recognition accuracy. We demonstrate this paradigm on recognition of unconstrained handwritten characters.
---
paper_title: Learning trajectories with a hierarchy of oscillatory modules

paper_content:

The most successful approach to learning has been the backpropagation method. Although very powerful on relatively simple problems, theoretical analysis and simulations show that this approach breaks down as soon as sufficiently complex problems are considered. To overcome this fundamental limitation, a hierarchical and modular approach is suggested, directly inspired from biological networks, whereby a certain degree of structure is introduced in the learning system. This approach is applied to a simple example of trajectory learning of a semi-figure eight. >
---
paper_title: Hardware architecture of a neural net based retina implant for patients suffering from retinitis pigmentosa

paper_content:

This work describes the architecture and planned realization of components for a retina implant which includes a sensory device for image acquisition, biologically inspired neural nets for modeling and computation of receptive field functions, and a human interface for electrostimulation of ganglion cells. The system will be used to generate visual sensations based on captured visual images for patients with degenerated retina functions. Special CMOS circuitry will be developed for an integrated photodetector array with preprocessing according to the requirements for biology inspired spatio-temporal signal processing. Also, the development will include a flexible silicon multielectrode structure with microelectronic action potential generation for stimulation of intact retinal ganglion cells and a wireless communication interface linking the external retina encoder hardware and the implanted stimulator hardware.
---
paper_title: Task Decomposition Through Competition in a Modular Connectionist Architecture

paper_content:

A NOVEL MODULAR CONNECTIONIST ARCHITECTURE IS PRESENTED IN WHICH THE NETWORKS COMPOSING THE ARCHITECTURE COMPETE TO LEARN THE TRAINING PATTERNS. AN OUTCOME OF THE COMPETITION IS THAT DIFFERENT NETWORKS LEARN DIFFERENT TRAINING PATTERNS AND, THUS, LEARN TO COMPUTE DIFFERENT FUNCTIONS. THE ARCHITECTURE PERFORMS TASK DECOMPOSITION IN THE SENSE THAT IT LEARNS TO PARTITION A TASK INTO TWO OR MORE FUNCTIONALLY INDEPENDENT TASKS AND ALLO- CATES DISTINCT NETWORKS TO LEARN EACH TASK. IN ADDITION, THE ARCHITECTURE TENDS TO ALLOCATE TO EACH TASK THE NETWORK WHOSE TOPOLOGY IS MORE APPROPRI- ATE TO THAT TASK. THE ARCHITECTURE''S PERFORMANCE ON "WHAT" AND "WHERE" VISION TASKS IS PRESENTED AND COMPARED WITH THE PERFORMANCE OF TWO MULTI- LAYER NETWORKS. FINALLY, IT IS NOTED THAT FUNCTION DECOMPOSITION IS AN UNDERCONSTRAINED PROBLEM AND, THUS, DIFFERENT MODULAR ARCHITECTURES MAY DECOMPOSE A FUNCTION IN DIFFERENT WAYS. WE ARGUE THAT A DESIRABLE DECOM- POSITION CAN BE ACHIEVED IF THE ARCHITECTURE IS SUITABLY RESTRICTED IN THE TYPES OF FUNCTIONS THAT IT CAN COMPUTE. APPROPRIATE RESTRICTIONS CAN BE FOUND THROUGH THE APPLICATION OF DOMAIN KNOWLEDGE. A STRENGTH OF THE MOD- ULAR ARCHITECTURE IS THAT ITS STRUCTURE IS WELL-SUITED FOR INCORPORATING DOMAIN KNOWLEDGE.
---
paper_title: The functional logic of cortical connections

paper_content:

Patterns of anatomical connections in the visual cortex form the structural basis for segregating features of the visual image into separate cortical areas and for communication between these areas at all levels to produce a coherent percept. Such multi-stage integration may be a common strategy throughout the cortex for producing complex behaviour.
---
paper_title: A new neural network structure with cooperative modules

paper_content:

A new neural network structure is proposed in order to solve complex (large) classification problems. At first, the problem is divided into a number of simpler sub-problems using an unsupervised network. Then, each sub-problem is solved using one module of the proposed network. The new addition to this traditional "divide-and-conquer" structure is a voting layer above the outputs of the modules. In this layer, the modules cooperate in deciding the final decision by training every module on some additional classes: each class represents a group of classes of another module. During testing, the average "vote" of the modules for each other is calculated and multiplied by the classes output. This scheme gave higher accuracy and faster learning speed than the backpropagation scheme. A recursive hierarchy of this network can be built-up for more enhancement to the performance. >
---
paper_title: Voting schemes for cooperative neural network classifiers

paper_content:

Multiple neural network modules cooperating in taking a classification decision are modeled as multiple voters electing one candidate in a single ballot election assuming the availability of votes' preferences and intensities. All modules are considered as candidates as well as voters. Voting bids are the output-activations of the modules forming the cooperative modular structure. Different voting schemes are compared according to the accuracy of defining classification decision boundaries. A higher classification accuracy implies a better representation of the information available at different preferences (output values). The effect of the modules' voting power on the accuracy of the decision is studied and integrated in the network's design strategy.
---
paper_title: How Cortical Interconnectedness Varies with Network Size

paper_content:

When model neural networks are used to gain insight into how the brain might carry out its computations, comparisons between features of the network and those of the brain form an important basis for drawing conclusions about the network's relevance to brain function. The most significant features to be compared, of course, relate to behavior of the units. Another network property that would be useful to consider, however, is the extent to which units are interconnected and the law by which unit-unit connections scale as the network is made larger. The goal of this paper is to consider these questions for neocortex. The conclusion will be that neocortical neurons are rather sparsely interconnected each neuron receives direct synaptic input from fewer than 3% of its neighbors underlying the surrounding square millimeter of cortex and the extent of connectedness hardly changes for brains that range in size over about four orders of magnitude. These conclusions support the currently popular notion that the brain's circuits are highly modular and suggest that increased cortex size is mainly achieved by adding more modules.
---
paper_title: Modular Learning in Neural Networks: A Modularized Approach to Neural Network Classification

paper_content:

From the Publisher: ::: This important new work recognizes the advanced nature of today's artificial neural networks, uniquely emphasizing a modular approach to neural network learning. By breaking down the learning task into relatively independent parts of lower complexity, Modular Learning in Neural Networks demonstrates how neural network learning can be made more powerful and efficient. The book's modular approach, unlike the monolithic viewpoint, admits intermediary solution stages whose success can be independently verified, as in other engineering fields. Each stage can be evaluated before moving on to the subsequent one, and the reason for possible failures can be analyzed, ultimately leading to the improved development and engineering of applications. The modular approach also takes into account growing network complexity, reducing the difficulty of such inevitable problems as scaling and convergence. Modular Learning in Neural Networks' modular approach is also fully in step with important psychological and neurobiological research. Studies in developmental psychology demonstrate the incremental nature of human learning, in which the success of each stage is conditioned by the successful accomplishment of the previous stage, while neurobiology has depicted the human brain as a complex structure of cooperating modules. Modular Learning in Neural Networks covers the full range of conceivable approaches to the modularization of learning, including decomposition of learning into modules using supervised and unsupervised learning types; decomposition of the function to be mapped into linear and nonlinear parts; decomposition of the neural network to minimize harmful interferences between a large number of network parameters during learning; decomposition of the application task into subtasks that are learned separately; decomposition into a knowledge-based part and a learning part. The book attempts to show that modular learning based on these approaches is helpful in improving t
---
paper_title: Catastrophic interference in human motor learning

paper_content:

Biological sensorimotor systems are not static maps that transform input (sensory information) into output (motor behavior). Evidence from many lines of research suggests that their representations are plastic, experience-dependent entities. While this plasticity is essential for flexible behavior, it presents the nervous system with difficult organizational challenges. If the sensorimotor system adapts itself to perform well under one set of circumstances, will it then perform poorly when placed in an environment with different demands (negative transfer)? Will a later experience-dependent change undo the benefits of previous learning (catastrophic interference)? We explore the first question in a separate paper in this volume (Shadmehr et al. 1995). Here we present psychophysical and computational results that explore the question of catastrophic interference in the context of a dynamic motor learning task. Under some conditions, subjects show evidence of catastrophic interference. Under other conditions, however, subjects appear to be immune to its effects. These results suggest that motor learning can undergo a process of consolidation. Modular neural networks are well suited for the demands of learning multiple input/output mappings. By incorporating the notion of fast- and slow-changing connections into a modular architecture, we were able to account for the psychophysical results.
---
paper_title: Integrating the BSP400 neurocomputer with the MetaNet network environment

paper_content:

The authors describe the integration of hardware and software components into a single neural network environment. The building blocks described are the BSP400, a massively parallel neurocomputer, and MetaNet, a general neural network design tool. This research is a feasibility study for future developments in large-scale integration of computational resources. The objective is to obtain a neural simulation environment that supports easy development and efficient parallel calculation. Integration takes place using device drivers. Apart from handling communications between the components the primary task of the device driver is the mapping of a neural network definition onto the processors of the parallel hardware such as the BSP400. This mapping problem is caused by hardware constraints, such as maximum amount of connections and the topology of the processor network. The ultimate goal is an environment in which the user can program a single 'ideal virtual' machine, which in reality is assembled from many different computational resources. >
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: Learning trajectories with a hierarchy of oscillatory modules

paper_content:

The most successful approach to learning has been the backpropagation method. Although very powerful on relatively simple problems, theoretical analysis and simulations show that this approach breaks down as soon as sufficiently complex problems are considered. To overcome this fundamental limitation, a hierarchical and modular approach is suggested, directly inspired from biological networks, whereby a certain degree of structure is introduced in the learning system. This approach is applied to a simple example of trajectory learning of a semi-figure eight. >
---
paper_title: Modular connectionist structure for 100-word recognition

paper_content:

A modular neural structure is presented which is dedicated to the recognition of larger vocabularies. It contains several so-called scaly subnets, assembled into a compound network by neural glue elements. This architecture and the corresponding training scheme have been investigated for various network parameters during speaker-dependent 100-word recognition. Important simulation results were compared with a multilayer perceptron showing scaly input-to-hidden connections and with standard dynamic time warping. Under the criterion of high recognition rates along with very short reaction time, modular subnet assemblies provide for successful recognition of 100 words. They are also recommended for speaker-independent classification. >
---
paper_title: Task Decomposition Through Competition in a Modular Connectionist Architecture

paper_content:

A NOVEL MODULAR CONNECTIONIST ARCHITECTURE IS PRESENTED IN WHICH THE NETWORKS COMPOSING THE ARCHITECTURE COMPETE TO LEARN THE TRAINING PATTERNS. AN OUTCOME OF THE COMPETITION IS THAT DIFFERENT NETWORKS LEARN DIFFERENT TRAINING PATTERNS AND, THUS, LEARN TO COMPUTE DIFFERENT FUNCTIONS. THE ARCHITECTURE PERFORMS TASK DECOMPOSITION IN THE SENSE THAT IT LEARNS TO PARTITION A TASK INTO TWO OR MORE FUNCTIONALLY INDEPENDENT TASKS AND ALLO- CATES DISTINCT NETWORKS TO LEARN EACH TASK. IN ADDITION, THE ARCHITECTURE TENDS TO ALLOCATE TO EACH TASK THE NETWORK WHOSE TOPOLOGY IS MORE APPROPRI- ATE TO THAT TASK. THE ARCHITECTURE''S PERFORMANCE ON "WHAT" AND "WHERE" VISION TASKS IS PRESENTED AND COMPARED WITH THE PERFORMANCE OF TWO MULTI- LAYER NETWORKS. FINALLY, IT IS NOTED THAT FUNCTION DECOMPOSITION IS AN UNDERCONSTRAINED PROBLEM AND, THUS, DIFFERENT MODULAR ARCHITECTURES MAY DECOMPOSE A FUNCTION IN DIFFERENT WAYS. WE ARGUE THAT A DESIRABLE DECOM- POSITION CAN BE ACHIEVED IF THE ARCHITECTURE IS SUITABLY RESTRICTED IN THE TYPES OF FUNCTIONS THAT IT CAN COMPUTE. APPROPRIATE RESTRICTIONS CAN BE FOUND THROUGH THE APPLICATION OF DOMAIN KNOWLEDGE. A STRENGTH OF THE MOD- ULAR ARCHITECTURE IS THAT ITS STRUCTURE IS WELL-SUITED FOR INCORPORATING DOMAIN KNOWLEDGE.
---
paper_title: Optoelectronic implementation of neural networks

paper_content:

This article discusses the optical architectures and techniques in implementing neural networks as they apply to optical pattern recognition and optical information processing systems.
---
paper_title: A new neural network structure with cooperative modules

paper_content:

A new neural network structure is proposed in order to solve complex (large) classification problems. At first, the problem is divided into a number of simpler sub-problems using an unsupervised network. Then, each sub-problem is solved using one module of the proposed network. The new addition to this traditional "divide-and-conquer" structure is a voting layer above the outputs of the modules. In this layer, the modules cooperate in deciding the final decision by training every module on some additional classes: each class represents a group of classes of another module. During testing, the average "vote" of the modules for each other is calculated and multiplied by the classes output. This scheme gave higher accuracy and faster learning speed than the backpropagation scheme. A recursive hierarchy of this network can be built-up for more enhancement to the performance. >
---
paper_title: Training a 3-Node Neural Network is NP-Complete

paper_content:

We consider a 2-layer, 3-node, n-input neural network whose nodes compute linear threshold functions of their inputs. We show that it is NP-complete to decide whether there exist weights and thresholds for the three nodes of this network so that it will produce output consistent with a given set of training examples. We extend the result to other simple networks. This result suggests that those looking for perfect training algorithms cannot escape inherent computational difficulties just by considering only simple or very regular networks. It also suggests the importance, given a training problem, of finding an appropriate network and input encoding for that problem. It is left as an open problem to extend our result to nodes with non-linear functions such as sigmoids.
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: Learning internal representations by error propagation

paper_content:

This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: Modular Learning in Neural Networks: A Modularized Approach to Neural Network Classification

paper_content:

From the Publisher: ::: This important new work recognizes the advanced nature of today's artificial neural networks, uniquely emphasizing a modular approach to neural network learning. By breaking down the learning task into relatively independent parts of lower complexity, Modular Learning in Neural Networks demonstrates how neural network learning can be made more powerful and efficient. The book's modular approach, unlike the monolithic viewpoint, admits intermediary solution stages whose success can be independently verified, as in other engineering fields. Each stage can be evaluated before moving on to the subsequent one, and the reason for possible failures can be analyzed, ultimately leading to the improved development and engineering of applications. The modular approach also takes into account growing network complexity, reducing the difficulty of such inevitable problems as scaling and convergence. Modular Learning in Neural Networks' modular approach is also fully in step with important psychological and neurobiological research. Studies in developmental psychology demonstrate the incremental nature of human learning, in which the success of each stage is conditioned by the successful accomplishment of the previous stage, while neurobiology has depicted the human brain as a complex structure of cooperating modules. Modular Learning in Neural Networks covers the full range of conceivable approaches to the modularization of learning, including decomposition of learning into modules using supervised and unsupervised learning types; decomposition of the function to be mapped into linear and nonlinear parts; decomposition of the neural network to minimize harmful interferences between a large number of network parameters during learning; decomposition of the application task into subtasks that are learned separately; decomposition into a knowledge-based part and a learning part. The book attempts to show that modular learning based on these approaches is helpful in improving t
---
paper_title: A new neural network structure with cooperative modules

paper_content:

A new neural network structure is proposed in order to solve complex (large) classification problems. At first, the problem is divided into a number of simpler sub-problems using an unsupervised network. Then, each sub-problem is solved using one module of the proposed network. The new addition to this traditional "divide-and-conquer" structure is a voting layer above the outputs of the modules. In this layer, the modules cooperate in deciding the final decision by training every module on some additional classes: each class represents a group of classes of another module. During testing, the average "vote" of the modules for each other is calculated and multiplied by the classes output. This scheme gave higher accuracy and faster learning speed than the backpropagation scheme. A recursive hierarchy of this network can be built-up for more enhancement to the performance. >
---
paper_title: Cooperation of neural nets and task decomposition

paper_content:

In order to solve classification problems when using neural nets, it may be necessary to decompose the global task into a few subtasks, each being processed by one specialized network or module. The task is thus performed through a cooperation between all the modules. Such a decomposition may help to find a more efficient solution. The authors describe experiments on cooperating nets, in this case a task that can be totally decomposed by the user using a priori knowledge of the problem. The example considered clearly shows that building an architecture made of independent specialized units is not the most efficient way to perform successive processings of the data. Important improvements, both in classification and speed performances, may be obtained by training the different modules together. Another important point is that this modularity is the only way to build architectures which are suitable for complex tasks. >
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: Multilayer feedforward networks are universal approximators

paper_content:

Abstract This paper rigorously establishes that standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired degree of accuracy, provided sufficiently many hidden units are available. In this sense, multilayer feedforward networks are a class of universal approximators.
---
paper_title: Solving large-scale optimization problems by divide-and-conquer neural networks

paper_content:

The authors present a neural network algorithm that is based on a divide-and-conquer strategy for solving large-scale traveling-salesman problems (TSPs). This ad hoc cut-and-splice algorithm utilizes a two-layer network with a partitioned first layer and a combining second layer. The cities of a large-scale TSP are first divided into clusters or local sets. Each local city set is represented by a neural network in the first layer, which upon convergence yields the local minimum tour. The second layer optimizes the interconnections between the local sets, forming a global final tour. The regional global (or local) minima of each cluster can be combined to form the global minima of the overall energy functions. Results of the cut-and-splice algorithm are compared with those of the single-layer modified Hopefield-Tank neural network algorithm. The invariant property is confirmed in the sense that the sum of the energy fixed points is equal to that of the fixed points of its parts, and the sum is insensitive to the partitioning of the energy landscape. This means that the linear boundary perturbation created by the partitioning is relatively small as the quadratic area of the problem increases. >
---
paper_title: Efficient classification for multiclass problems using modular neural networks

paper_content:

The rate of convergence of net output error is very low when training feedforward neural networks for multiclass problems using the backpropagation algorithm. While backpropagation will reduce the Euclidean distance between the actual and desired output vectors, the differences between some of the components of these vectors increase in the first iteration. Furthermore, the magnitudes of subsequent weight changes in each iteration are very small, so that many iterations are required to compensate for the increased error in some components in the initial iterations. Our approach is to use a modular network architecture, reducing a K-class problem to a set of K two-class problems, with a separately trained network for each of the simpler problems. Speedups of one order of magnitude have been obtained experimentally, and in some cases convergence was possible using the modular approach but not using a nonmodular network. >
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: Task Decomposition Through Competition in a Modular Connectionist Architecture

paper_content:

A NOVEL MODULAR CONNECTIONIST ARCHITECTURE IS PRESENTED IN WHICH THE NETWORKS COMPOSING THE ARCHITECTURE COMPETE TO LEARN THE TRAINING PATTERNS. AN OUTCOME OF THE COMPETITION IS THAT DIFFERENT NETWORKS LEARN DIFFERENT TRAINING PATTERNS AND, THUS, LEARN TO COMPUTE DIFFERENT FUNCTIONS. THE ARCHITECTURE PERFORMS TASK DECOMPOSITION IN THE SENSE THAT IT LEARNS TO PARTITION A TASK INTO TWO OR MORE FUNCTIONALLY INDEPENDENT TASKS AND ALLO- CATES DISTINCT NETWORKS TO LEARN EACH TASK. IN ADDITION, THE ARCHITECTURE TENDS TO ALLOCATE TO EACH TASK THE NETWORK WHOSE TOPOLOGY IS MORE APPROPRI- ATE TO THAT TASK. THE ARCHITECTURE''S PERFORMANCE ON "WHAT" AND "WHERE" VISION TASKS IS PRESENTED AND COMPARED WITH THE PERFORMANCE OF TWO MULTI- LAYER NETWORKS. FINALLY, IT IS NOTED THAT FUNCTION DECOMPOSITION IS AN UNDERCONSTRAINED PROBLEM AND, THUS, DIFFERENT MODULAR ARCHITECTURES MAY DECOMPOSE A FUNCTION IN DIFFERENT WAYS. WE ARGUE THAT A DESIRABLE DECOM- POSITION CAN BE ACHIEVED IF THE ARCHITECTURE IS SUITABLY RESTRICTED IN THE TYPES OF FUNCTIONS THAT IT CAN COMPUTE. APPROPRIATE RESTRICTIONS CAN BE FOUND THROUGH THE APPLICATION OF DOMAIN KNOWLEDGE. A STRENGTH OF THE MOD- ULAR ARCHITECTURE IS THAT ITS STRUCTURE IS WELL-SUITED FOR INCORPORATING DOMAIN KNOWLEDGE.
---
paper_title: Vapnik-Chervonenkis Dimension Bounds for Two- and Three-Layer Networks

paper_content:

We show that the Vapnik-Chervonenkis dimension of the class of functions that can be computed by arbitrary two-layer or some completely connected three-layer threshold networks with real inputs is at least linear in the number of weights in the network. In Valiant's "probably approximately correct" learning framework, this implies that the number of random training examples necessary for learning in these networks is at least linear in the number of weights.
---
paper_title: On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities

paper_content:

This chapter reproduces the English translation by B. Seckler of the paper by Vapnik and Chervonenkis in which they gave proofs for the innovative results they had obtained in a draft form in July 1966 and announced in 1968 in their note in Soviet Mathematics Doklady. The paper was first published in Russian as Вапник В. Н. and Червоненкис А. Я. О равномерноЙ сходимости частот появления событиЙ к их вероятностям. Теория вероятностеЙ и ее применения 16(2), 264–279 (1971).
---
paper_title: The Vapnik-Chervonenkis Dimension: Information versus Complexity in Learning

paper_content:

When feasible, learning is a very attractive alternative to explicit programming. This is particularly true in areas where the problems do not lend themselves to systematic programming, such as pattern recognition in natural environments. The feasibility of learning an unknown function from examples depends on two questions: ::: ::: 1. Do the examples convey enough information to determine the function? ::: ::: 2. Is there a speedy way of constructing the function from the examples? ::: ::: These questions contrast the roles of information and complexity in learning. While the two roles share some ground, they are conceptually and technically different. In the common language of learning, the information question is that of generalization and the complexity question is that of scaling. The work of Vapnik and Chervonenkis (1971) provides the key tools for dealing with the information issue. In this review, we develop the main ideas of this framework and discuss how complexity fits in.
---
paper_title: Pattern Discrimination Using Feedforward Networks: A Benchmark Study of Scaling Behavior

paper_content:

The discrimination powers of multilayer perceptron (MLP) and learning vector quantization (LVQ) networks are compared for overlapping gaussian distributions. It is shown, both analytically and with Monte Carlo studies, that the MLP network handles high-dimensional problems in a more efficient way than LVQ. This is mainly due to the sigmoidal form of the MLP transfer function, but also to the fact that the MLP uses hyperplanes more efficiently. Both algorithms are equally robust to limited training sets and the learning curves fall off like 1/M, where M is the training set size, which is compared to theoretical predictions from statistical estimates and Vapnik-Chervonenkis bounds.
---
paper_title: A Modular and Hybrid Connectionist System for Speaker Identification

paper_content:

This paper presents and evaluates a modular/hybrid connectionist system for speaker identification. Modularity has emerged as a powerful technique for reducing the complexity of connectionist systems, and allowing a priori knowledge to be incorporated into their design. Text-independent speaker identification is an inherently complex task where the amount of training data is often limited. It thus provides an ideal domain to test the validity of the modular/hybrid connectionist approach. To achieve such identification, we develop, in this paper, an architecture based upon the cooperation of several connectionist modules, and a Hidden Markov Model module. When tested on a population of 102 speakers extracted from the DARPA-TIMIT database, perfect identification was obtained.
---
paper_title: An adaptive and fully sparse training approach for multilayer perceptrons

paper_content:

An adaptive and fully sparse backpropagation training approach is proposed in this paper. The technique speeds up training by combining a sparse optimization concept with neural network training. The sparse phenomenon due to neuron activation, which is inherent in neural networks, is exploited in both feedforward and backpropagation phases. A new computational algorithm with sparse pattern reuse and refreshment has been developed together with the adaptation procedure of a new set of parameters which regulate the sparse training process. The proposed training approach has been applied to speech recognition and circuit extraction problems and achieved significant speed-up of training.
---
paper_title: Optimal Brain Damage

paper_content:

We have used information-theoretic ideas to derive a class of practical and nearly optimal schemes for adapting the size of a neural network. By removing unimportant weights from a network, several improvements can be expected: better generalization, fewer training examples required, and improved speed of learning and/or classification. The basic idea is to use second-derivative information to make a tradeoff between network complexity and training set error. Experiments confirm the usefulness of the methods on a real-world application.
---
paper_title: Links Between Artificial Neural Networks (ANN) and Statistical Pattern Recognition

paper_content:

Publisher Summary This chapter discusses the ways in which artificial neural networks (ANNs) differ from the well-known paradigms of statistical pattern recognition (SPR) and discusses whether there are any concepts in ANN for which no counterpart in SPR exists and vice-versa. The chapter explains the benefits that can come out of interaction between ANN and SPR researchers. It defines the advantages that ANN techniques have over SPR methods in dealing with real-world problems. For the most part, ANNs for pattern recognition are to SPR what poetry is to prose— technically a subset, based on the acceptance of certain design constraints of enormous practical utility, plus an additional source of inspiration that can be very helpful in the design process. Beyond that, the underlying concepts and paradigms are so close and the range of problems under study overlap to such a degree that greater mutual communication can minimize the duplication of effort and enrich both communities in many other ways. One area of substantive difference is that some parts of the ANN community treat pattern recognition as a subset of systems to perform dynamic system identification and control; this can lead to improvements in pattern recognition, in some cases, when the pattern to be recognized is essentially dynamic.
---
paper_title: Task Decomposition Through Competition in a Modular Connectionist Architecture

paper_content:

A NOVEL MODULAR CONNECTIONIST ARCHITECTURE IS PRESENTED IN WHICH THE NETWORKS COMPOSING THE ARCHITECTURE COMPETE TO LEARN THE TRAINING PATTERNS. AN OUTCOME OF THE COMPETITION IS THAT DIFFERENT NETWORKS LEARN DIFFERENT TRAINING PATTERNS AND, THUS, LEARN TO COMPUTE DIFFERENT FUNCTIONS. THE ARCHITECTURE PERFORMS TASK DECOMPOSITION IN THE SENSE THAT IT LEARNS TO PARTITION A TASK INTO TWO OR MORE FUNCTIONALLY INDEPENDENT TASKS AND ALLO- CATES DISTINCT NETWORKS TO LEARN EACH TASK. IN ADDITION, THE ARCHITECTURE TENDS TO ALLOCATE TO EACH TASK THE NETWORK WHOSE TOPOLOGY IS MORE APPROPRI- ATE TO THAT TASK. THE ARCHITECTURE''S PERFORMANCE ON "WHAT" AND "WHERE" VISION TASKS IS PRESENTED AND COMPARED WITH THE PERFORMANCE OF TWO MULTI- LAYER NETWORKS. FINALLY, IT IS NOTED THAT FUNCTION DECOMPOSITION IS AN UNDERCONSTRAINED PROBLEM AND, THUS, DIFFERENT MODULAR ARCHITECTURES MAY DECOMPOSE A FUNCTION IN DIFFERENT WAYS. WE ARGUE THAT A DESIRABLE DECOM- POSITION CAN BE ACHIEVED IF THE ARCHITECTURE IS SUITABLY RESTRICTED IN THE TYPES OF FUNCTIONS THAT IT CAN COMPUTE. APPROPRIATE RESTRICTIONS CAN BE FOUND THROUGH THE APPLICATION OF DOMAIN KNOWLEDGE. A STRENGTH OF THE MOD- ULAR ARCHITECTURE IS THAT ITS STRUCTURE IS WELL-SUITED FOR INCORPORATING DOMAIN KNOWLEDGE.
---
paper_title: Subgrouping Reduces Complexity and Speeds Up Learning in Recurrent Networks

paper_content:

Recurrent nets are more powerful than feedforward nets because they allow simulation of dynamical systems. Everything from sine wave generators through computers to the brain are potential candidates, but to use recurrent nets to emulate dynamical systems we need learning algorithms to program them. Here I describe a new twist on an old algorithm for recurrent nets and compare it to its predecessors.
---
paper_title: Automatic target recognition using image and network decomposition

paper_content:

A constrained version of the general automatic target recognition problem is formulated in the context of pattern recognition, as the detection and localization of target patterns in scene images. An approach to the simplification of the learning involved is presented. The two-stage network architecture that results from this approach incorporates two cascaded modules: a multilayered perceptron followed by a modified Maxnet. The network provides translation-invariant recognition of targets. Simulation results are presented. >
---
paper_title: Cooperation of neural nets and task decomposition

paper_content:

In order to solve classification problems when using neural nets, it may be necessary to decompose the global task into a few subtasks, each being processed by one specialized network or module. The task is thus performed through a cooperation between all the modules. Such a decomposition may help to find a more efficient solution. The authors describe experiments on cooperating nets, in this case a task that can be totally decomposed by the user using a priori knowledge of the problem. The example considered clearly shows that building an architecture made of independent specialized units is not the most efficient way to perform successive processings of the data. Important improvements, both in classification and speed performances, may be obtained by training the different modules together. Another important point is that this modularity is the only way to build architectures which are suitable for complex tasks. >
---
paper_title: Improving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks

paper_content:

When either detection rate (sensitivity) or false alarm rate (specificity) is optimized in an artificial neural network trained to identify myocardial infarction, the increase in the accuracy of one is always done at the expense of the accuracy of the other. To overcome this loss, two networks that were separately trained on populations of patients with different likelihoods of myocardial infarction were used in concert. One network was trained on clinical pattern sets derived from patients who had a low likelihood of myocardial infarction, while the other was trained on pattern sets derived from patients with a high likelihood of myocardial infarction. Unknown patterns were analyzed by both networks. If the output generated by the network trained on the low risk patients was below an empirically set threshold, this output was chosen as the diagnostic output. If the output was above that threshold, the output of the network trained on the high risk patients was used as the diagnostic output. The dual network correctly identified 39 of the 40 patients who had sustained a myocardial infarction and 301 of 306 patients who did not have a myocardial infarction for a detection rate (sensitivity) and false alarm rate (1-specificity) of 97.50 and 1.63%, respectively. A parallel control experiment using a single network but identical training information correctly identified 39 of 40 patients who had sustained a myocardial infarction and 287 of 306 patients who had not sustained a myocardial infarction (p = 0.003).
---
paper_title: Modular Learning in Neural Networks: A Modularized Approach to Neural Network Classification

paper_content:

From the Publisher: ::: This important new work recognizes the advanced nature of today's artificial neural networks, uniquely emphasizing a modular approach to neural network learning. By breaking down the learning task into relatively independent parts of lower complexity, Modular Learning in Neural Networks demonstrates how neural network learning can be made more powerful and efficient. The book's modular approach, unlike the monolithic viewpoint, admits intermediary solution stages whose success can be independently verified, as in other engineering fields. Each stage can be evaluated before moving on to the subsequent one, and the reason for possible failures can be analyzed, ultimately leading to the improved development and engineering of applications. The modular approach also takes into account growing network complexity, reducing the difficulty of such inevitable problems as scaling and convergence. Modular Learning in Neural Networks' modular approach is also fully in step with important psychological and neurobiological research. Studies in developmental psychology demonstrate the incremental nature of human learning, in which the success of each stage is conditioned by the successful accomplishment of the previous stage, while neurobiology has depicted the human brain as a complex structure of cooperating modules. Modular Learning in Neural Networks covers the full range of conceivable approaches to the modularization of learning, including decomposition of learning into modules using supervised and unsupervised learning types; decomposition of the function to be mapped into linear and nonlinear parts; decomposition of the neural network to minimize harmful interferences between a large number of network parameters during learning; decomposition of the application task into subtasks that are learned separately; decomposition into a knowledge-based part and a learning part. The book attempts to show that modular learning based on these approaches is helpful in improving t
---
paper_title: Modular connectionist structure for 100-word recognition

paper_content:

A modular neural structure is presented which is dedicated to the recognition of larger vocabularies. It contains several so-called scaly subnets, assembled into a compound network by neural glue elements. This architecture and the corresponding training scheme have been investigated for various network parameters during speaker-dependent 100-word recognition. Important simulation results were compared with a multilayer perceptron showing scaly input-to-hidden connections and with standard dynamic time warping. Under the criterion of high recognition rates along with very short reaction time, modular subnet assemblies provide for successful recognition of 100 words. They are also recommended for speaker-independent classification. >
---
paper_title: Modular Construction of Time-Delay Neural Networks for Speech Recognition

paper_content:

Several strategies are described that overcome limitations of basic network models as steps towards the design of large connectionist speech recognition systems. The two major areas of concern are the problem of time and the problem of scaling. Speech signals continuously vary over time and encode and transmit enormous amounts of human knowledge. To decode these signals, neural networks must be able to use appropriate representations of time and it must be possible to extend these nets to almost arbitrary sizes and complexity within finite resources. The problem of time is addressed by the development of a Time-Delay Neural Network; the problem of scaling by Modularity and Incremental Design of large nets based on smaller subcomponent nets. It is shown that small networks trained to perform limited tasks develop time invariant, hidden abstractions that can subsequently be exploited to train larger, more complex nets efficiently. Using these techniques, phoneme recognition networks of increasing complexity...
---
paper_title: Computing truck attributes with artificial neural networks

paper_content:

The paper describes the application of neural networks to the problem of determining truck attributes (such as velocity, axle spacings, and axle loads) purely from strain‐response readings taken from the structure over which the truck is traveling. The approach is designed to remove both the need for tape switches on the deck of the bridge to obtain such data and associated problems so as to provide a convenient and viable means of collecting bridge loading statistics. The application and performance of a radial‐Gaussian‐based networking system with its own training algorithm to the truck‐attribute determination problem is detailed. The chosen approach is a two‐layered modular network structure. The artificial neural network in the first layer classifies the trucks, and the second‐layer artificial neural networks—three for each type of truck—compute estimates of the truck's velocity, axle spacings, and axle loads. This solution provides a fast, accurate, and convenient means of determining truck attributes.
---
paper_title: A large phonemic time-delay neural network technique for all Mandarin consonants recognition

paper_content:

A large phonemic time-delay neural network (LPTDNN) technique has been developed to recognize all Mandarin consonants of the entire set of Chinese syllables. By modular construction, six component networks, viz. (b,d,g,ot), (l,m,n,r,ot), (j,z,zh,ot), (c,q,ch,ot), (f,s,x,h,sh,ot) and (p,t,k,ot), are formed to represent all 21 Mandarin consonants (where ot=others). The technique was thoroughly tested with 8 sets of 1157 isolated Hanyu Pinyin syllables, with 6 sets used for training and 2 sets used for testing. The overall result shows a high recognition performance with 99.2% for the inside test (i.e. testing on the datasets which were used entirely in training) and 92.7% for the outside test (i.e. testing on the remaining datasets which were never used in training). >
---
paper_title: A modular architecture for efficient learning

paper_content:

A modular architecture for supervised learning is presented. Three ways to modularize the learning are investigated: (1) preprocessing the input by a self-organizing subnetwork to extract strong features from the data, (2) supervised feature extraction, and (3) an iterative learning cycle in which only one layer learns at a time, with the output-layer weights learned by an exact method. With this modular architecture, only a small fraction of connection weights is determined by the gradient-descent method. A series of computational experiments shows the superiority of the modular model in learning quality and speed. The author begins by considering the decomposition of a feedforward network into a feature-discovery and a supervised-learning modules, then he introduces supervised feature discovery, and finally he describes a modular backpropagation algorithm
---
paper_title: Modular Learning in Neural Networks: A Modularized Approach to Neural Network Classification

paper_content:

From the Publisher: ::: This important new work recognizes the advanced nature of today's artificial neural networks, uniquely emphasizing a modular approach to neural network learning. By breaking down the learning task into relatively independent parts of lower complexity, Modular Learning in Neural Networks demonstrates how neural network learning can be made more powerful and efficient. The book's modular approach, unlike the monolithic viewpoint, admits intermediary solution stages whose success can be independently verified, as in other engineering fields. Each stage can be evaluated before moving on to the subsequent one, and the reason for possible failures can be analyzed, ultimately leading to the improved development and engineering of applications. The modular approach also takes into account growing network complexity, reducing the difficulty of such inevitable problems as scaling and convergence. Modular Learning in Neural Networks' modular approach is also fully in step with important psychological and neurobiological research. Studies in developmental psychology demonstrate the incremental nature of human learning, in which the success of each stage is conditioned by the successful accomplishment of the previous stage, while neurobiology has depicted the human brain as a complex structure of cooperating modules. Modular Learning in Neural Networks covers the full range of conceivable approaches to the modularization of learning, including decomposition of learning into modules using supervised and unsupervised learning types; decomposition of the function to be mapped into linear and nonlinear parts; decomposition of the neural network to minimize harmful interferences between a large number of network parameters during learning; decomposition of the application task into subtasks that are learned separately; decomposition into a knowledge-based part and a learning part. The book attempts to show that modular learning based on these approaches is helpful in improving t
---
paper_title: What Size Net Gives Valid Generalization?

paper_content:

We address the question of when a network can be expected to generalize from m random training examples chosen from some arbitrary probability distribution, assuming that future test examples are drawn from the same distribution. Among our results are the following bounds on appropriate sample vs. network size. Assume 0 < e ≤ 1/8. We show that if m ≥ O(W/e log N/e) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 - e/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 1 - e of future test examples drawn from the same distribution. Conversely, for fully-connected feedforward nets with one hidden layer, any learning algorithm using fewer than Ω(W/e) random training examples will, for some distributions of examples consistent with an appropriate weight choice, fail at least some fixed fraction of the time to find a weight choice that will correctly classify more than a 1 - e fraction of the future test examples.
---
paper_title: Cooperation of neural nets for robust classification

paper_content:

The problem of building robust neural network classification algorithms that perform well on a wide variety of problems is addressed. Evidence of the fact that current procedures are only accurate on a restricted range of tasks is presented. The authors propose multimodule architectures based on the cooperation of two or more neural net techniques as a solution. This idea is illustrated by the cooperation of a multilayer perceptron and a learning vector quantization algorithm. It is shown that this method combines the advantages of its individual components and is much more robust. It is accurate for a large range of problems and is easy to tune. An algorithm that allows the direct training of this multimodule architecture is described. The use of this technique enhances performance when dealing with classification problems
---
paper_title: Modular learning in neural networks

paper_content:

In the development of large-scale knowledge networks much recent progress has been inspired by connections to neurobiology. An important component of any "neural" network is an accompanying learning algorithm. Such an algorithm, to be biologically plausible, must work for very large numbers of units. Studies of large-scale systems have so far been restricted to systems Without internal units (units With no direct connections to the input or output). Internal units are crucial to such systems as they are the means by which a system can encode high-order regularities (or invariants) that are Implicit in its inputs and outputs. Computer simulations of learning using internal units have been restricted to small-scale systems. This paper describes away of coupling autoassociative learning modules Into hierarchies that should greatly improve the performance of learning algorithms in large-scale systems. The Idea has been tested experimentally with positive results.
---
paper_title: Analysis of input vector space for speeding up learning in feedforward neural networks

paper_content:

Although several new learning rules have been proposed, the space and time complexities of these two bottlenecks remain. The bottlenecks occur because all the existing learning techniques are based on a single large neural structure. In this paper, the authors approach the solution to this problem by analyzing the input vector space and, then, partition this space into subspaces. Each subspace will be learned by a small neural network with a simple learning rule. The authors also speed up the learning process by reducing the number of training vectors down to O(mn) instead of 2/sup n/, where m is the number of vectors having output targets equal to ones and n is the number of input bits. The reduction is based on the concept of guard ring vectors. The experimental results show that the learning can be speeded up by 2-30 times over the non-partition process. However the number of neurons used in the authors' approach is uncontrollable. In some cases, the number is reduced but in some cases it is increased. >
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: Modular connectionist structure for 100-word recognition

paper_content:

A modular neural structure is presented which is dedicated to the recognition of larger vocabularies. It contains several so-called scaly subnets, assembled into a compound network by neural glue elements. This architecture and the corresponding training scheme have been investigated for various network parameters during speaker-dependent 100-word recognition. Important simulation results were compared with a multilayer perceptron showing scaly input-to-hidden connections and with standard dynamic time warping. Under the criterion of high recognition rates along with very short reaction time, modular subnet assemblies provide for successful recognition of 100 words. They are also recommended for speaker-independent classification. >
---
paper_title: Modular Neural Network Classifiers: A Comparative Study

paper_content:

There is a wide variety of Modular Neural Network (MNN) classifiers in the literature. They differ according to the design of their architecture, task-decomposition scheme, learning procedure, and multi-module decision-making strategy. Meanwhile, there is a lack of comparative studies in the MNN literature. This paper compares ten MNN classifiers which give a good representation of design varieties, viz., Decoupled; Other-output; ART-BP; Hierarchical; Multiple-experts; Ensemble (majority vote); Ensemble (average vote); Merge-glue; Hierarchical Competitive Neural Net; and Cooperative Modular Neural Net. Two benchmark applications of different degree and nature of complexity are used for performance comparison, and the strength-points and drawbacks of the different networks are outlined. The aim is to help a potential user to choose an appropriate model according to the application in hand and the available computational resources.
---
paper_title: Classification of hand-written digits by a large scale neural network 'CombNET-II'

paper_content:

The authors describe the network structure and the learning procedure of CombNET-II together with experimental results on hand-written digit classification. CombNET-II uses the self-growing neural network learning procedure for training the stem network. After training the stem network, all input data are partitioned into category groups. Then, branch networks are trained for every category group. Backpropagation is utilized to train branch networks. Each branch neural network which is a three-layered hierarchical network has only a small number of connections so that it is easy to tune up. Therefore, CombNET-II has good convergency in the learning process. CombNET-II has been applied to the classification of 6349 printed Kanji characters and to the recognition of 1000 spoken words. CombNET-II correctly classified 99.4% of previously unseen hand-written digits. >
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: Ensemble pruning algorithms for accelerated training

paper_content:

The error surface on which minimization is done in any feedforward neural network training algorithm is highly irregular, with multiple local minima having been observed empirically. In training schemes, this implies that several random initial points must be chosen, and the performance of the resulting trained neural network evaluated for each such choice, in order to obtain a well-trained network. However, training is computationally expensive, and often one may have a limit on the number of training cycles allowed during training, thereby making the total number of cycles required to find the best-trained net too large for this brute-force method to be practical. It is therefore desirable to find an algorithm which eliminates "bad" networks during training itself, without utilizing the full allowed number of training cycles, and in such a way as to minimize the average total training cycles. We present two such algorithms which are easy to implement.
---
paper_title: A new neural network structure with cooperative modules

paper_content:

A new neural network structure is proposed in order to solve complex (large) classification problems. At first, the problem is divided into a number of simpler sub-problems using an unsupervised network. Then, each sub-problem is solved using one module of the proposed network. The new addition to this traditional "divide-and-conquer" structure is a voting layer above the outputs of the modules. In this layer, the modules cooperate in deciding the final decision by training every module on some additional classes: each class represents a group of classes of another module. During testing, the average "vote" of the modules for each other is calculated and multiplied by the classes output. This scheme gave higher accuracy and faster learning speed than the backpropagation scheme. A recursive hierarchy of this network can be built-up for more enhancement to the performance. >
---
paper_title: A dynamic neural network architecture by sequential partitioning of the input space

paper_content:

A sequential approach to neural network training is presented. The network is presented with each item of data only once, and its architecture is dynamically adjusted during training. At the arrival of each example, a decision whether to increase the complexity of the network is made, based on three heuristic criteria. These criteria measure the position of the new item of data with respect to the information currently stored in the network. Each hidden unit in the network is trained in closed form by means of a recursive least squares algorithm. A local covariance matrix of the data is maintained at each node, and the closed form solution is recursively updated. The performance of the algorithm is illustrated on a small-scale problem, using two-dimensional speech data. The sequential nature of the algorithm has an efficient hardware implementation in the form of systolic arrays, and the incremental training idea has better biological plausibility. >
---
paper_title: Solving large-scale optimization problems by divide-and-conquer neural networks

paper_content:

The authors present a neural network algorithm that is based on a divide-and-conquer strategy for solving large-scale traveling-salesman problems (TSPs). This ad hoc cut-and-splice algorithm utilizes a two-layer network with a partitioned first layer and a combining second layer. The cities of a large-scale TSP are first divided into clusters or local sets. Each local city set is represented by a neural network in the first layer, which upon convergence yields the local minimum tour. The second layer optimizes the interconnections between the local sets, forming a global final tour. The regional global (or local) minima of each cluster can be combined to form the global minima of the overall energy functions. Results of the cut-and-splice algorithm are compared with those of the single-layer modified Hopefield-Tank neural network algorithm. The invariant property is confirmed in the sense that the sum of the energy fixed points is equal to that of the fixed points of its parts, and the sum is insensitive to the partitioning of the energy landscape. This means that the linear boundary perturbation created by the partitioning is relatively small as the quadratic area of the problem increases. >
---
paper_title: A divide-and-conquer methodology for modular supervised neural network design

paper_content:

A novel learning strategy based on the divide-and-conquer concept is proposed to effectively overcome the slow learning speed and hard-determined network size problems in supervised learning neural networks. The proposed method first partitions the whole complex training set into several manageable subsets and then generates small size networks to 'conquer' (or learn) all these training subsets. In order to achieve efficient partition on a train set, we have proposed an error correlation partitioning (ECP) scheme such that sub-training-sets are formed with small (acceptable) training error. Based on this learning strategy, a self-growing modular neural network system can be developed. By applying the proposed learning strategy, a neural network is not only useful for pattern classification problems but also for continuous valued function approximation problems. >
---
paper_title: Analysis of input vector space for speeding up learning in feedforward neural networks

paper_content:

Although several new learning rules have been proposed, the space and time complexities of these two bottlenecks remain. The bottlenecks occur because all the existing learning techniques are based on a single large neural structure. In this paper, the authors approach the solution to this problem by analyzing the input vector space and, then, partition this space into subspaces. Each subspace will be learned by a small neural network with a simple learning rule. The authors also speed up the learning process by reducing the number of training vectors down to O(mn) instead of 2/sup n/, where m is the number of vectors having output targets equal to ones and n is the number of input bits. The reduction is based on the concept of guard ring vectors. The experimental results show that the learning can be speeded up by 2-30 times over the non-partition process. However the number of neurons used in the authors' approach is uncontrollable. In some cases, the number is reduced but in some cases it is increased. >
---
paper_title: Classification of hand-written digits by a large scale neural network 'CombNET-II'

paper_content:

The authors describe the network structure and the learning procedure of CombNET-II together with experimental results on hand-written digit classification. CombNET-II uses the self-growing neural network learning procedure for training the stem network. After training the stem network, all input data are partitioned into category groups. Then, branch networks are trained for every category group. Backpropagation is utilized to train branch networks. Each branch neural network which is a three-layered hierarchical network has only a small number of connections so that it is easy to tune up. Therefore, CombNET-II has good convergency in the learning process. CombNET-II has been applied to the classification of 6349 printed Kanji characters and to the recognition of 1000 spoken words. CombNET-II correctly classified 99.4% of previously unseen hand-written digits. >
---
paper_title: Improving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks

paper_content:

When either detection rate (sensitivity) or false alarm rate (specificity) is optimized in an artificial neural network trained to identify myocardial infarction, the increase in the accuracy of one is always done at the expense of the accuracy of the other. To overcome this loss, two networks that were separately trained on populations of patients with different likelihoods of myocardial infarction were used in concert. One network was trained on clinical pattern sets derived from patients who had a low likelihood of myocardial infarction, while the other was trained on pattern sets derived from patients with a high likelihood of myocardial infarction. Unknown patterns were analyzed by both networks. If the output generated by the network trained on the low risk patients was below an empirically set threshold, this output was chosen as the diagnostic output. If the output was above that threshold, the output of the network trained on the high risk patients was used as the diagnostic output. The dual network correctly identified 39 of the 40 patients who had sustained a myocardial infarction and 301 of 306 patients who did not have a myocardial infarction for a detection rate (sensitivity) and false alarm rate (1-specificity) of 97.50 and 1.63%, respectively. A parallel control experiment using a single network but identical training information correctly identified 39 of 40 patients who had sustained a myocardial infarction and 287 of 306 patients who had not sustained a myocardial infarction (p = 0.003).
---
paper_title: A modular architecture for efficient learning

paper_content:

A modular architecture for supervised learning is presented. Three ways to modularize the learning are investigated: (1) preprocessing the input by a self-organizing subnetwork to extract strong features from the data, (2) supervised feature extraction, and (3) an iterative learning cycle in which only one layer learns at a time, with the output-layer weights learned by an exact method. With this modular architecture, only a small fraction of connection weights is determined by the gradient-descent method. A series of computational experiments shows the superiority of the modular model in learning quality and speed. The author begins by considering the decomposition of a feedforward network into a feature-discovery and a supervised-learning modules, then he introduces supervised feature discovery, and finally he describes a modular backpropagation algorithm
---
paper_title: Multiple networks for function learning

paper_content:

In the case of learning a mapping, it is proposed to build several possible models instead of one, train them all independently on the same task and take a vote over their responses. These networks will converge to different solutions due to using different models, different parameter set sizes or any other factor related to training. Two training methods are used, i.e., grow and learn (GAL), a memory based method, and backpropagation. Several voting schemes are investigated, and their performances are composed in the case of classification on a real world application (recognition of handwritten numerals) and a two-dimensional didactic case. The weights in voting may be interpreted in two ways: the certainty of a network in its output, and in a Bayesian setting, the plausibility, i.e., the prior probability of the model. In all cases tested, the result of voting is better than the results of all of the networks that participated in the voting process. >
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: Modular Learning in Neural Networks: A Modularized Approach to Neural Network Classification

paper_content:

From the Publisher: ::: This important new work recognizes the advanced nature of today's artificial neural networks, uniquely emphasizing a modular approach to neural network learning. By breaking down the learning task into relatively independent parts of lower complexity, Modular Learning in Neural Networks demonstrates how neural network learning can be made more powerful and efficient. The book's modular approach, unlike the monolithic viewpoint, admits intermediary solution stages whose success can be independently verified, as in other engineering fields. Each stage can be evaluated before moving on to the subsequent one, and the reason for possible failures can be analyzed, ultimately leading to the improved development and engineering of applications. The modular approach also takes into account growing network complexity, reducing the difficulty of such inevitable problems as scaling and convergence. Modular Learning in Neural Networks' modular approach is also fully in step with important psychological and neurobiological research. Studies in developmental psychology demonstrate the incremental nature of human learning, in which the success of each stage is conditioned by the successful accomplishment of the previous stage, while neurobiology has depicted the human brain as a complex structure of cooperating modules. Modular Learning in Neural Networks covers the full range of conceivable approaches to the modularization of learning, including decomposition of learning into modules using supervised and unsupervised learning types; decomposition of the function to be mapped into linear and nonlinear parts; decomposition of the neural network to minimize harmful interferences between a large number of network parameters during learning; decomposition of the application task into subtasks that are learned separately; decomposition into a knowledge-based part and a learning part. The book attempts to show that modular learning based on these approaches is helpful in improving t
---
paper_title: Cooperation of neural nets for robust classification

paper_content:

The problem of building robust neural network classification algorithms that perform well on a wide variety of problems is addressed. Evidence of the fact that current procedures are only accurate on a restricted range of tasks is presented. The authors propose multimodule architectures based on the cooperation of two or more neural net techniques as a solution. This idea is illustrated by the cooperation of a multilayer perceptron and a learning vector quantization algorithm. It is shown that this method combines the advantages of its individual components and is much more robust. It is accurate for a large range of problems and is easy to tune. An algorithm that allows the direct training of this multimodule architecture is described. The use of this technique enhances performance when dealing with classification problems
---
paper_title: Modular learning in neural networks

paper_content:

In the development of large-scale knowledge networks much recent progress has been inspired by connections to neurobiology. An important component of any "neural" network is an accompanying learning algorithm. Such an algorithm, to be biologically plausible, must work for very large numbers of units. Studies of large-scale systems have so far been restricted to systems Without internal units (units With no direct connections to the input or output). Internal units are crucial to such systems as they are the means by which a system can encode high-order regularities (or invariants) that are Implicit in its inputs and outputs. Computer simulations of learning using internal units have been restricted to small-scale systems. This paper describes away of coupling autoassociative learning modules Into hierarchies that should greatly improve the performance of learning algorithms in large-scale systems. The Idea has been tested experimentally with positive results.
---
paper_title: ENSEMBLE COMPETITIVE LEARNING NEURAL NETWORKS WITH REDUCED INPUT DIMENSION

paper_content:

Conventional neural networks utilize all the dimensions of the original input patterns for training and classification. However, a particular attribute of the input patterns does not necessarily contribute to classification and may even cause misclassification in certain cases. A new ensemble competitive learning method using the reduced input dimension is proposed. In contrast to the previous ensemble neural networks which adjust learning parameters, the proposed method takes advantage of the information in each dimension of the input patterns. Since the degree of contribution of each attribute to classification is not known beforehand, the different input data sets with one dimension reduced are presented to multiple neural networks. The classification information from each competitive learning neural network is then combined to make a final decision for classification. In order to improve classification accuracy, the ambiguous output neurons are eliminated which cannot be assigned to any class after training. We use three consensus schemes to judge the classification using ensemble neural networks. The experimental results with remote sensing and speech data indicate the improved performance of the proposed method.
---
paper_title: NN/II: improved version of a network for large-scale pattern recognition tasks

paper_content:

A network architecture NN/I which divides and learns environments was previously proposed for pattern recognition. NN/I divides a task of learning input-output relations into several subtasks adaptively, and then learns each subtask in a part of the network, i.e., subnetwork, by means of the error backpropagation algorithm. The results of an application of NN/I to recognizing handwritten Japanese characters are summarized, and its faults in generalization ability revealed in the application are analyzed. NN/II, which accommodates various novel devices to remove the faults in NN/I, is introduced, and its satisfactory ability is demonstrated through an application to character recognition. >
---
paper_title: Democracy in neural nets: voting schemes for classification

paper_content:

Abstract In this paper we discuss some possible ways to combine the outputs of a set of neural network classifiers to reach a combined decision with a higher performance, in terms of lower rejection rates and/or better accuracy rates. The methods considered range from the requirement of a complete agreement among the individual classifications to election schemes based on the distribution of votes collected by the different classes. In addition, the rejection rules based on the different output classes can be complemented by rules that also consider the information in the individual output vectors, with the possibility of using threshold requirements and that of averaging the different vectors. Although the Bayesian framework and some probabilistic assumptions provide useful indications about the potential advantage of different combination schemes, the combined performance ultimately depends on the joint probability distribution of the outputs, and it can be estimated by joining the results of different nets on the same test set. The combination methods are very flexible, they permit a straightforward cooperation of neural and traditional recognizers, and they are appropriate in a development environment where experiments are performed with different kinds of nets and features for a selected application. From our experiments in the field of handwritten digit recognition (up to a total of more than 50,000 characters), we found that the use of a small number of nets (two to three) with a sufficiently large uncorrelation in their mistakes reaches a combined performance that is significantly higher than the best obtainable from the individual nets, with a negligible effort after starting from a pool of networks produced in the development phase of an application. In particular, for a real-world OCR application, the best accuracy increase is about half the increase in the rejection rate, so that accuracies of the order of 99.5% can be reached by rejecting less than 5% of the patterns. This performance is significant for real applications.
---
paper_title: Asymptotical analysis of a modular neural network

paper_content:

Modular neural networks have been used in several applications because of their superiority over a single neural network in terms of faster learning, proper data representation, and feasibility of hardware implementation. This paper presents an asymptotical performance analysis showing that the performance of a modular neural network is always better than or as good as that of a single neural network when both neural networks are optimized. The minimum mean square error (MSE) that can be achieved by a modular neural network is also obtained.
---
paper_title: A Large-Scale Neural Network Which Recognizes Handwritten Kanji Characters

paper_content:

We propose a new way to construct a large-scale neural network for 3.000 handwritten Kanji characters recognition. This neural network consists of 3 parts: a collection of small-scale networks which are trained individually on a small number of Kanji characters; a network which integrates the output from the small-scale networks, and a process to facilitate the integration of these neworks. The recognition rate of the total system is comparable with those of the small-scale networks. Our results indicate that the proposed method is effective for constructing a large-scale network without loss of recognition performance.
---
paper_title: Modular Construction of Time-Delay Neural Networks for Speech Recognition

paper_content:

Several strategies are described that overcome limitations of basic network models as steps towards the design of large connectionist speech recognition systems. The two major areas of concern are the problem of time and the problem of scaling. Speech signals continuously vary over time and encode and transmit enormous amounts of human knowledge. To decode these signals, neural networks must be able to use appropriate representations of time and it must be possible to extend these nets to almost arbitrary sizes and complexity within finite resources. The problem of time is addressed by the development of a Time-Delay Neural Network; the problem of scaling by Modularity and Incremental Design of large nets based on smaller subcomponent nets. It is shown that small networks trained to perform limited tasks develop time invariant, hidden abstractions that can subsequently be exploited to train larger, more complex nets efficiently. Using these techniques, phoneme recognition networks of increasing complexity...
---
paper_title: A new neural network structure with cooperative modules

paper_content:

A new neural network structure is proposed in order to solve complex (large) classification problems. At first, the problem is divided into a number of simpler sub-problems using an unsupervised network. Then, each sub-problem is solved using one module of the proposed network. The new addition to this traditional "divide-and-conquer" structure is a voting layer above the outputs of the modules. In this layer, the modules cooperate in deciding the final decision by training every module on some additional classes: each class represents a group of classes of another module. During testing, the average "vote" of the modules for each other is calculated and multiplied by the classes output. This scheme gave higher accuracy and faster learning speed than the backpropagation scheme. A recursive hierarchy of this network can be built-up for more enhancement to the performance. >
---
paper_title: Cooperation of neural nets and task decomposition

paper_content:

In order to solve classification problems when using neural nets, it may be necessary to decompose the global task into a few subtasks, each being processed by one specialized network or module. The task is thus performed through a cooperation between all the modules. Such a decomposition may help to find a more efficient solution. The authors describe experiments on cooperating nets, in this case a task that can be totally decomposed by the user using a priori knowledge of the problem. The example considered clearly shows that building an architecture made of independent specialized units is not the most efficient way to perform successive processings of the data. Important improvements, both in classification and speed performances, may be obtained by training the different modules together. Another important point is that this modularity is the only way to build architectures which are suitable for complex tasks. >
---
paper_title: Improving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks

paper_content:

When either detection rate (sensitivity) or false alarm rate (specificity) is optimized in an artificial neural network trained to identify myocardial infarction, the increase in the accuracy of one is always done at the expense of the accuracy of the other. To overcome this loss, two networks that were separately trained on populations of patients with different likelihoods of myocardial infarction were used in concert. One network was trained on clinical pattern sets derived from patients who had a low likelihood of myocardial infarction, while the other was trained on pattern sets derived from patients with a high likelihood of myocardial infarction. Unknown patterns were analyzed by both networks. If the output generated by the network trained on the low risk patients was below an empirically set threshold, this output was chosen as the diagnostic output. If the output was above that threshold, the output of the network trained on the high risk patients was used as the diagnostic output. The dual network correctly identified 39 of the 40 patients who had sustained a myocardial infarction and 301 of 306 patients who did not have a myocardial infarction for a detection rate (sensitivity) and false alarm rate (1-specificity) of 97.50 and 1.63%, respectively. A parallel control experiment using a single network but identical training information correctly identified 39 of 40 patients who had sustained a myocardial infarction and 287 of 306 patients who had not sustained a myocardial infarction (p = 0.003).
---
paper_title: Classification of hand-written digits by a large scale neural network 'CombNET-II'

paper_content:

The authors describe the network structure and the learning procedure of CombNET-II together with experimental results on hand-written digit classification. CombNET-II uses the self-growing neural network learning procedure for training the stem network. After training the stem network, all input data are partitioned into category groups. Then, branch networks are trained for every category group. Backpropagation is utilized to train branch networks. Each branch neural network which is a three-layered hierarchical network has only a small number of connections so that it is easy to tune up. Therefore, CombNET-II has good convergency in the learning process. CombNET-II has been applied to the classification of 6349 printed Kanji characters and to the recognition of 1000 spoken words. CombNET-II correctly classified 99.4% of previously unseen hand-written digits. >
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: NN/II: improved version of a network for large-scale pattern recognition tasks

paper_content:

A network architecture NN/I which divides and learns environments was previously proposed for pattern recognition. NN/I divides a task of learning input-output relations into several subtasks adaptively, and then learns each subtask in a part of the network, i.e., subnetwork, by means of the error backpropagation algorithm. The results of an application of NN/I to recognizing handwritten Japanese characters are summarized, and its faults in generalization ability revealed in the application are analyzed. NN/II, which accommodates various novel devices to remove the faults in NN/I, is introduced, and its satisfactory ability is demonstrated through an application to character recognition. >
---
paper_title: Improving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks

paper_content:

When either detection rate (sensitivity) or false alarm rate (specificity) is optimized in an artificial neural network trained to identify myocardial infarction, the increase in the accuracy of one is always done at the expense of the accuracy of the other. To overcome this loss, two networks that were separately trained on populations of patients with different likelihoods of myocardial infarction were used in concert. One network was trained on clinical pattern sets derived from patients who had a low likelihood of myocardial infarction, while the other was trained on pattern sets derived from patients with a high likelihood of myocardial infarction. Unknown patterns were analyzed by both networks. If the output generated by the network trained on the low risk patients was below an empirically set threshold, this output was chosen as the diagnostic output. If the output was above that threshold, the output of the network trained on the high risk patients was used as the diagnostic output. The dual network correctly identified 39 of the 40 patients who had sustained a myocardial infarction and 301 of 306 patients who did not have a myocardial infarction for a detection rate (sensitivity) and false alarm rate (1-specificity) of 97.50 and 1.63%, respectively. A parallel control experiment using a single network but identical training information correctly identified 39 of 40 patients who had sustained a myocardial infarction and 287 of 306 patients who had not sustained a myocardial infarction (p = 0.003).
---
paper_title: Construction of a large-scale neural network: simulation of handwritten Japanese character recognition on NCUBE

paper_content:

This paper describes how new learning methods may make it possible for a large-scale, hierarchical neural network to recognize most Japanese handwritten characters. This is a very large and complex task, as the Japanese character set consists of about 3000 categories which can be written in many different ways. Such a difficult task can lead a neural network to converge very slowly and to yield recognition rates that are uneven between categories. To address these problems we here propose five learning methods as modifications of the conventional back-propagation learning rule. These methods produce fast convergence, even recognition rates over all categories, and adequate recognition of test samples. We also describe how a large-scale neural network can be built by dividing the recognition task into several subtasks, with networks for each subtask, and then integrating these subnetworks in a large network with a hierarchical structure. In a hierarchical network, the upper level network directly integrates outputs from each lower level network. Application of that network to handwritten Japanese character recognition has resulted in poor recognition, because lower level networks do not know about unknown input patterns, and the direct integration of ambiguous outputs from many lower level networks confuses the upper level network. We propose a new integration method which provides each subnetwork with more information as to how close an input pattern is to the categories of that subnetwork. This method resulted in high recognition performance for character recognition. We here described the above methods, and report the performance of our implementation of a neural network for the recognition of 71 Hiragana characters, and describe our implementation of this network on a hypercube concurrent computer.
---
paper_title: A large phonemic time-delay neural network technique for all Mandarin consonants recognition

paper_content:

A large phonemic time-delay neural network (LPTDNN) technique has been developed to recognize all Mandarin consonants of the entire set of Chinese syllables. By modular construction, six component networks, viz. (b,d,g,ot), (l,m,n,r,ot), (j,z,zh,ot), (c,q,ch,ot), (f,s,x,h,sh,ot) and (p,t,k,ot), are formed to represent all 21 Mandarin consonants (where ot=others). The technique was thoroughly tested with 8 sets of 1157 isolated Hanyu Pinyin syllables, with 6 sets used for training and 2 sets used for testing. The overall result shows a high recognition performance with 99.2% for the inside test (i.e. testing on the datasets which were used entirely in training) and 92.7% for the outside test (i.e. testing on the remaining datasets which were never used in training). >
---
paper_title: Multiple networks for function learning

paper_content:

In the case of learning a mapping, it is proposed to build several possible models instead of one, train them all independently on the same task and take a vote over their responses. These networks will converge to different solutions due to using different models, different parameter set sizes or any other factor related to training. Two training methods are used, i.e., grow and learn (GAL), a memory based method, and backpropagation. Several voting schemes are investigated, and their performances are composed in the case of classification on a real world application (recognition of handwritten numerals) and a two-dimensional didactic case. The weights in voting may be interpreted in two ways: the certainty of a network in its output, and in a Bayesian setting, the plausibility, i.e., the prior probability of the model. In all cases tested, the result of voting is better than the results of all of the networks that participated in the voting process. >
---
paper_title: Neural network ensembles

paper_content:

Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. >
---
paper_title: Democracy in neural nets: voting schemes for classification

paper_content:

Abstract In this paper we discuss some possible ways to combine the outputs of a set of neural network classifiers to reach a combined decision with a higher performance, in terms of lower rejection rates and/or better accuracy rates. The methods considered range from the requirement of a complete agreement among the individual classifications to election schemes based on the distribution of votes collected by the different classes. In addition, the rejection rules based on the different output classes can be complemented by rules that also consider the information in the individual output vectors, with the possibility of using threshold requirements and that of averaging the different vectors. Although the Bayesian framework and some probabilistic assumptions provide useful indications about the potential advantage of different combination schemes, the combined performance ultimately depends on the joint probability distribution of the outputs, and it can be estimated by joining the results of different nets on the same test set. The combination methods are very flexible, they permit a straightforward cooperation of neural and traditional recognizers, and they are appropriate in a development environment where experiments are performed with different kinds of nets and features for a selected application. From our experiments in the field of handwritten digit recognition (up to a total of more than 50,000 characters), we found that the use of a small number of nets (two to three) with a sufficiently large uncorrelation in their mistakes reaches a combined performance that is significantly higher than the best obtainable from the individual nets, with a negligible effort after starting from a pool of networks produced in the development phase of an application. In particular, for a real-world OCR application, the best accuracy increase is about half the increase in the rejection rate, so that accuracies of the order of 99.5% can be reached by rejecting less than 5% of the patterns. This performance is significant for real applications.
---
paper_title: Modular neural network architectures for classification

paper_content:

One of the major drawbacks of the current neural network generation is the inability to cope with the increase of size/complexity of classification tasks. Modular neural network classifiers attempt to solve this problem through a "divide and conquer" approach. However. The performance of the modular neural network classifiers is sensitive to efficiency of the "task decomposition" technique and the "multi-module decision-making" strategy. After a brief review of previous work with emphasis on five published modular classifiers-decoupled nets, ART-BP, hierarchical network, multiple experts, and multiple identical networks (majority vote and average output decisions)-this paper introduces the cooperative modular neural network (CMNN). The CMNN classifier outperforms the surveyed nets due to its novel task decomposition and multi-module decision-making techniques.
---
paper_title: A Neuro-Fuzzy Architecture for High Performance Classification

paper_content:

The concept of combining modular neural networks has been recently exploited as a new direction for the development of highly reliable neural network systems in the area of pattern classification. In this paper we present an efficient method for combining the modular networks based on fuzzy logic, especially the fuzzy integral. This method nonlinearly combines objective evidences, in the form of network outputs, with subjective evaluation of the reliability of the individual neural networks. Also, for more effective aggregation, we adopt the extension of the fuzzy integral with ordered weighted averaging operators. The experimental results with the recognition problem of on-line handwriting characters show that the performance of individual networks could be improved significantly.
---
paper_title: Voting schemes for cooperative neural network classifiers

paper_content:

Multiple neural network modules cooperating in taking a classification decision are modeled as multiple voters electing one candidate in a single ballot election assuming the availability of votes' preferences and intensities. All modules are considered as candidates as well as voters. Voting bids are the output-activations of the modules forming the cooperative modular structure. Different voting schemes are compared according to the accuracy of defining classification decision boundaries. A higher classification accuracy implies a better representation of the information available at different preferences (output values). The effect of the modules' voting power on the accuracy of the decision is studied and integrated in the network's design strategy.
---

Title: Modular Neural Networks: A Survey

Section 1: Introduction
Description 1: This section introduces the topic of Modular Neural Networks (MNN), describing their structure, history, and the motivations driving their development.

Section 2: Motivations
Description 2: This section discusses various motivations for developing MNNs, including biological, psychological, hardware, and computational reasons.

Section 3: Biological Motivations
Description 3: This section explores the biological inspirations behind MNNs, including modularity in biological systems and the functional specialization of brain regions.

Section 4: Psychological Motivations
Description 4: This section examines the psychological aspects that drive the modularization of NN learning algorithms and structures.

Section 5: Hardware Motivations
Description 5: This section analyzes the hardware constraints and advantages of using MNNs over nonmodular NNs, particularly concerning implementation and scalability.

Section 6: Complexity of Learning
Description 6: This section delves into the difficulties and solutions related to the learning performance of NNs, particularly focusing on the coupling of hidden nodes and strategies to mitigate it.

Section 7: NN Size
Description 7: This section discusses the impact of network size on the performance and efficiency of NNs, and how modularization can aid in determining the appropriate size.

Section 8: Generalization
Description 8: This section highlights the importance of generalization in NNs and how modularization can improve their ability to generalize from training data to real-world applications.

Section 9: Speed of Learning
Description 9: This section deals with the speed of learning in NNs and how modular structures can enhance learning speed by reducing crosstalk and facilitating parallel processing.

Section 10: MNN Design Stages
Description 10: This section outlines the general steps in designing MNNs, which include task decomposition, training, and multimodule decision-making.

Section 11: Task Decomposition
Description 11: This section describes various methods for decomposing tasks into sub-tasks for different modules within an MNN.

Section 12: Training Modules
Description 12: This section discusses strategies for training the different modules of an MNN, whether sequentially or in parallel.

Section 13: Multimodule Decision-Making
Description 13: This section explores how decisions from different modules can be integrated into a final global decision using competitive or cooperative strategies.

Section 14: MNNs in Real Applications
Description 14: This section provides examples of how MNNs have been successfully applied in real-world scenarios and their advantages over nonmodular NNs.

Section 15: Conclusions
Description 15: This section summarizes the findings of the survey and provides recommendations for future research and design of MNNs.