A Survey of Fuzzy Clustering Algorithms for Pattern Recognition - Part 11
---
paper_title: Self-organizing neural network as a fuzzy classifier

paper_content:

This paper describes a self-organizing artificial neural network, based on Kohonen's model of self-organization, which is capable of handling fuzzy input and of providing fuzzy classification. Unlike conventional neural net models, this algorithm incorporates fuzzy set-theoretic concepts at various stages. The input vector consists of membership values for linguistic properties along with some contextual class membership information which is used during self-organization to permit efficient modeling of fuzzy (ambiguous) patterns. A new definition of gain factor for weight updating is proposed. An index of disorder involving mean square distance between the input and weight vectors is used to determine a measure of the ordering of the output space. This controls the number of sweeps required in the process. Incorporation of the concept of fuzzy partitioning allows natural self-organization of the input data, especially when they have ill-defined boundaries. The output of unknown test patterns is generated in terms of class membership values. Incorporation of fuzziness in input and output is seen to provide better performance as compared to the original Kohonen model and the hard version. The effectiveness of this algorithm is demonstrated on the speech recognition problem for various network array sizes, training sets and gain factors. >
---
paper_title: Fuzzy clustering: critical analysis of the contextual mechanisms employed by three neural network models

paper_content:

According to the following definition, taken from the literature, a fuzzy clustering mechanism allows the same input pattern to belong to multiple categories to different degrees. Many clustering neural network (NN) models claim to feature fuzzy properties, but several of them (like the Fuzzy ART model) do not satisfy this definition. Vice versa, we believe that Kohonen's Self-Organizing Map, SOM, satisfies the definition provided above, even though this NN model is well-known to (robustly) perform topologically ordered mapping rather than fuzzy clustering. This may sound as a paradox if we consider that several fuzzy NN models (such as the Fuzzy Learning Vector Quantization, FLVQ, which was first called Fuzzy Kohonen Clustering Network, FKCN) were originally developed to enhance Kohonen's models (such as SOM and the vector quantization model, VQ). The fuzziness of SOM indicates that a network of processing elements (PEs) can verify the fuzzy clustering definition when it exploits local rules which are biologically plausible (such as the Kohonen bubble strategy). This is equivalent to state that the exploitation of the fuzzy set theory in the development of complex systems (e.g., clustering NNs) may provide new mathematical tools (e.g., the definition of membership function) to simulate the behavior of those cooperative/competitive mechanisms already identified by neurophysiological studies. When a biologically plausible cooperative/competitive strategy is pursued effectively, neighboring PEs become mutually coupled to gain sensitivity to contextual effects. PEs which are mutually coupled are affected by vertical (inter-layer) as well as horizontal (intra-layer) connections. To summarize, we suggest to relate the study of fuzzy clustering mechanisms to the multi-disciplinary science of complex systems, with special regard to the investigation of the cooperative/competitive local rules employed by complex systems to gain sensitivity to contextual effects in cognitive tasks. In this paper, the FLVQ model is critically analyzed in order to stress the meaning of a fuzzy learning mechanism. This study leads to the development of a new NN model, termed the fuzzy competitive/cooperative Kohonen (FCCK) model, which replaces FLVQ. Then, the architectural differences amongst three NN algorithms and the relationships between their fuzzy clustering properties are discussed. These models, which all perform on-line learning, are: (1) SOM; (2) FCCK; and (3) improved neural-gas (INC).
---
paper_title: On the importance of sorting in "neural gas" training of vector quantizers

paper_content:

The paper considers the role of the sorting process in the well-known "neural gas" model for vector quantization. Theoretical derivations and experimental evidence show that complete sorting is not required for effective training, since limiting the sorted list to even a few top units performs effectively. This property has a significant impact on the implementation of the overall neural model at the local level.
---
paper_title: A possibilistic approach to clustering

paper_content:

The clustering problem is cast in the framework of possibility theory. The approach differs from the existing clustering methods in that the resulting partition of the data can be interpreted as a possibilistic partition, and the membership values can be interpreted as degrees of possibility of the points belonging to the classes, i.e., the compatibilities of the points with the class prototypes. An appropriate objective function whose minimum will characterize a good possibilistic partition of the data is constructed, and the membership and prototype update equations are derived from necessary conditions for minimization of the criterion function. The advantages of the resulting family of possibilistic algorithms are illustrated by several examples. >
---
paper_title: Comments on "A possibilistic approach to clustering"

paper_content:

In this comment, we report a difficulty with the-application of the possibilistic approach to fuzzy clustering (PCM) proposed by Keller and Krishnapuram (1993). In applying this algorithm we found that it has the undesirable tendency to produce coincidental clusters. Results illustrating this tendency are reported and a possible explanation for the PCM behavior is suggested.
---
paper_title: Fuzzy Kohonen clustering networks

paper_content:

The authors propose a fuzzy Kohonen clustering network which integrates the fuzzy c-means (FCM) model into the learning rate and updating strategies of the Kohonen network. This yields an optimization problem related to FCM, and the numerical results show improved convergence as well as reduced labeling errors. It is proved that the proposed scheme is equivalent to the c-means algorithms. The new method can be viewed as a Kohonen type of FCM, but it is self-organizing, since the size of the update neighborhood and the learning rate in the competitive layer are automatically adjusted during learning. Anderson's IRIS data were used to illustrate this method. The results are compared with the standard Kohonen approach. >
---
paper_title: A Constructive, Incremental-Learning Network for Mixture Modeling and Classification

paper_content:

Gaussian ARTMAP (GAM) is a supervised-learning adaptive resonance theory (ART) network that uses gaussian-defined receptive fields. Like other ART networks, GAM incrementally learns and constructs a representation of sufficient complexity to solve a problem it is trained on. GAM's representation is a gaussian mixture model of the input space, with learned mappings from the mixture components to output classes. We show a close relationship between GAM and the well-known expectation-maximization (EM) approach to mixture modeling. GAM outper forms an EM classification algorithm on three classification benchmarks, thereby demonstrating the advantage of the ART match criterion for regulating learning and the ARTMAP match tracking operation for incorporating environmental feedback in supervised learning situations.
---
paper_title: A possibilistic approach to clustering

paper_content:

The clustering problem is cast in the framework of possibility theory. The approach differs from the existing clustering methods in that the resulting partition of the data can be interpreted as a possibilistic partition, and the membership values can be interpreted as degrees of possibility of the points belonging to the classes, i.e., the compatibilities of the points with the class prototypes. An appropriate objective function whose minimum will characterize a good possibilistic partition of the data is constructed, and the membership and prototype update equations are derived from necessary conditions for minimization of the criterion function. The advantages of the resulting family of possibilistic algorithms are illustrated by several examples. >
---
paper_title: Generalized clustering networks and Kohonen's self-organizing scheme

paper_content:

The relationship between the sequential hard c-means (SHCM) and learning vector quantization (LVQ) clustering algorithms is discussed. The impact and interaction of these two families of methods with Kohonen's self-organizing feature mapping (SOFM), which is not a clustering method but often lends ideas to clustering algorithms, are considered. A generalization of LVQ that updates all nodes for a given input vector is proposed. The network attempts to find a minimum of a well-defined objective function. The learning rules depend on the degree of distance match to the winner node; the lesser the degree of match with the winner, the greater the impact on nonwinner nodes. Numerical results indicate that the terminal prototypes generated by this modification of LVQ are generally insensitive to initialization and independent of any choice of learning coefficient. IRIS data obtained by E. Anderson's (1939) is used to illustrate the proposed method. Results are compared with the standard LVQ approach. >
---
paper_title: Gaussian ARTMAP: a neural network for fast incremental learning of noisy multidimensional maps

paper_content:

A new neural network architecture for incremental supervised learning of analog multidimensional maps is introduced. The architecture, called Gaussian ARTMAP, is a synthesis of a Gaussian classifier and an adaptive resonance theory (ART) neural network, achieved by defining the ART choice function as the discriminant function of a Gaussian classifier with separable distributions, and the ART match function as the same, but with the distributions normalized to a unit height. While Gaussian ARTMAP retains the attractive parallel computing and fast learning properties of fuzzy ARTMAP, it learns a more efficient internal representation of a mapping while being more resistant to noise than fuzzy ARTMAP on a number of benchmark databases. SSeveral simulations are presented which demonstrate that Gaussian ARTMAP consistently obtains a better trade-off of classification rate to number of categories than fuzzy ARTMAP. Results on a vowel classification problem are also presented which demonstrate that Gaussian ARTMAP outperforms many other classifiers. Copyright © 1996 Elsevier Science Ltd
---
paper_title: ;Neural-gas' network for vector quantization and its application to time-series prediction.

paper_content:

A neural network algorithm based on a soft-max adaptation rule is presented. This algorithm exhibits good performance in reaching the optimum minimization of a cost function for vector quantization data compression. The soft-max rule employed is an extension of the standard K-means clustering procedure and takes into account a neighborhood ranking of the reference (weight) vectors. It is shown that the dynamics of the reference (weight) vectors during the input-driven adaptation procedure are determined by the gradient of an energy function whose shape can be modulated through a neighborhood determining parameter and resemble the dynamics of Brownian particles moving in a potential determined by the data point density. The network is used to represent the attractor of the Mackey-Glass equation and to predict the Mackey-Glass time series, with additional local linear mappings for generating output values. The results obtained for the time-series prediction compare favorably with the results achieved by backpropagation and radial basis function networks.
---
paper_title: A Tutorial on Support Vector Machines for Pattern Recognition

paper_content:

The tutorial starts with an overview of the concepts of VC dimension and structural risk minimization. We then describe linear Support Vector Machines (SVMs) for separable and non-separable data, working through a non-trivial example in detail. We describe a mechanical analogy, and discuss when SVM solutions are unique and when they are global. We describe how support vector training can be practically implemented, and discuss in detail the kernel mapping technique which is used to construct SVM solutions which are nonlinear in the data. We show how Support Vector machines can have very large (even infinite) VC dimension by computing the VC dimension for homogeneous polynomial and Gaussian radial basis function kernels. While very high VC dimension would normally bode ill for generalization performance, and while at present there exists no theory which shows that good generalization performance is guaranteed for SVMs, there are several arguments which support the observed high accuracy of SVMs, which we review. Results of some experiments which were inspired by these arguments are also presented. We give numerous examples and proofs of most of the key theorems. There is new material, and I hope that the reader will find that even old material is cast in a fresh light.
---
paper_title: Competitive Learning Algorithms for Robust Vector Quantization

paper_content:

The efficient representation and encoding of signals with limited resources, e.g., finite storage capacity and restricted transmission bandwidth, is a fundamental problem in technical as well as biological information processing systems. Typically, under realistic circumstances, the encoding and communication of messages has to deal with different sources of noise and disturbances. We propose a unifying approach to data compression by robust vector quantization, which explicitly deals with channel noise, bandwidth limitations, and random elimination of prototypes. The resulting algorithm is able to limit the detrimental effect of noise in a very general communication scenario. In addition, the presented model allows us to derive a novel competitive neural networks algorithm, which covers topology preserving feature maps, the so-called neural-gas algorithm, and the maximum entropy soft-max rule as special cases. Furthermore, continuation methods based on these noise models improve the codebook design by reducing the sensitivity to local minima. We show an exemplary application of the novel robust vector quantization algorithm to image compression for a teleconferencing system.
---
paper_title: Optimal learning in artificial neural networks: A review of theoretical results

paper_content:

Abstract The effectiveness of connectionist models in emulating intelligent behaviour and solving significant practical problems is strictly related to the capability of the learning algorithms to find optimal or near-optimal solutions and to generalize to new examples. This paper reviews some theoretical contributions to optimal learning in the attempt to provide a unified view and give the state of the art in the field. The focus of the review is on the problem of local minima in the cost function that is likely to affect more or less any learning algorithm. Starting from this analysis, we briefly review proposals for discovering optimal solutions and suggest conditions for designing architectures tailored to a given task.
---
paper_title: Novel neural network model combining radial basis function, competitive Hebbian learning rule, and fuzzy simplified adaptive resonance theory

paper_content:

In the first part of this paper a new on-line fully self- organizing artificial neural network model (FSONN), pursuing dynamic generation and removal of neurons and synaptic links, is proposed. The model combines properties of the self- organizing map (SOM), fuzzy c-means (FCM), growing neural gas (GNG) and fuzzy simplified adaptive resonance theory (Fuzzy SART) algorithms. In the second part of the paper experimental results are provided and discussed. Our conclusion is that the proposed connectionist model features several interesting properties, such as the following: (1) the system requires no a priori knowledge of the dimension, size and/or adjacency structure of the network; (2) with respect to other connectionist models found in the literature, the system can be employed successfully in: (a) a vector quantization; (b) density function estimation; and (c) structure detection in input data to be mapped topologically correctly onto an output lattice pursuing dimensionality reduction; and (3) the system is computationally efficient, its processing time increasing linearly with the number of neurons and synaptic links.
---
paper_title: Complex Systems and Cognitive Processes

paper_content:

This book shows that the science of complex systems, which stresses the importance of self-organizing processes, can make a decisive contribution to solving many problems in artificial intelligence. Artificial cognitive systems are important in view of their potential applications, and it can be expected that their study will shed light on biological cognitive systems. The new "neurally inspired" information science proposed in this book is fast becoming a promising workshop for the construction of models capable of emulating cognitive behaviour. After a general introduction to the theory of complex systems, the book gives a thorough treatment of neural networks, which are the most successful and the most thoroughly studied dynamical cognitive systems. Attention is also devoted to other classes of artificial cognitive systems, in particular to classifier systems, which provide an important link between the dynamical and the inferential approach to artificial intelligence. The book can be used as a textbook, since it does not require previous knowledge of the topic, and should also be interesting for researchers in this field, since it links formerly separate lines of research.
---
paper_title: Learning internal representations by error propagation

paper_content:

This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion
---
paper_title: The self-organizing map

paper_content:

The self-organized map, an architecture suggested for artificial neural networks, is explained by presenting simulation experiments and practical applications. The self-organizing map has the property of effectively creating spatially organized internal representations of various features of input signals and their abstractions. One result of this is that the self-organization process can discover semantic relationships in sentences. Brain maps, semantic maps, and early work on competitive learning are reviewed. The self-organizing map algorithm (an algorithm which order responses spatially) is reviewed, focusing on best matching cell selection and adaptation of the weight vectors. Suggestions for applying the self-organizing map algorithm, demonstrations of the ordering process, and an example of hierarchical clustering of data are presented. Fine tuning the map by learning vector quantization is addressed. The use of self-organized maps in practical speech recognition and a simulation experiment on semantic mapping are discussed. >
---
paper_title: ;Neural-gas' network for vector quantization and its application to time-series prediction.

paper_content:

A neural network algorithm based on a soft-max adaptation rule is presented. This algorithm exhibits good performance in reaching the optimum minimization of a cost function for vector quantization data compression. The soft-max rule employed is an extension of the standard K-means clustering procedure and takes into account a neighborhood ranking of the reference (weight) vectors. It is shown that the dynamics of the reference (weight) vectors during the input-driven adaptation procedure are determined by the gradient of an energy function whose shape can be modulated through a neighborhood determining parameter and resemble the dynamics of Brownian particles moving in a potential determined by the data point density. The network is used to represent the attractor of the Mackey-Glass equation and to predict the Mackey-Glass time series, with additional local linear mappings for generating output values. The results obtained for the time-series prediction compare favorably with the results achieved by backpropagation and radial basis function networks.
---
paper_title: Optimal learning in artificial neural networks: A review of theoretical results

paper_content:

Abstract The effectiveness of connectionist models in emulating intelligent behaviour and solving significant practical problems is strictly related to the capability of the learning algorithms to find optimal or near-optimal solutions and to generalize to new examples. This paper reviews some theoretical contributions to optimal learning in the attempt to provide a unified view and give the state of the art in the field. The focus of the review is on the problem of local minima in the cost function that is likely to affect more or less any learning algorithm. Starting from this analysis, we briefly review proposals for discovering optimal solutions and suggest conditions for designing architectures tailored to a given task.
---
paper_title: A Tutorial on Support Vector Machines for Pattern Recognition

paper_content:

The tutorial starts with an overview of the concepts of VC dimension and structural risk minimization. We then describe linear Support Vector Machines (SVMs) for separable and non-separable data, working through a non-trivial example in detail. We describe a mechanical analogy, and discuss when SVM solutions are unique and when they are global. We describe how support vector training can be practically implemented, and discuss in detail the kernel mapping technique which is used to construct SVM solutions which are nonlinear in the data. We show how Support Vector machines can have very large (even infinite) VC dimension by computing the VC dimension for homogeneous polynomial and Gaussian radial basis function kernels. While very high VC dimension would normally bode ill for generalization performance, and while at present there exists no theory which shows that good generalization performance is guaranteed for SVMs, there are several arguments which support the observed high accuracy of SVMs, which we review. Results of some experiments which were inspired by these arguments are also presented. We give numerous examples and proofs of most of the key theorems. There is new material, and I hope that the reader will find that even old material is cast in a fresh light.
---
paper_title: The Nature of Statistical Learning Theory

paper_content:

Setting of the learning problem consistency of learning processes bounds on the rate of convergence of learning processes controlling the generalization ability of learning processes constructing learning algorithms what is important in learning theory?.
---
paper_title: Optimal learning in artificial neural networks: A review of theoretical results

paper_content:

Abstract The effectiveness of connectionist models in emulating intelligent behaviour and solving significant practical problems is strictly related to the capability of the learning algorithms to find optimal or near-optimal solutions and to generalize to new examples. This paper reviews some theoretical contributions to optimal learning in the attempt to provide a unified view and give the state of the art in the field. The focus of the review is on the problem of local minima in the cost function that is likely to affect more or less any learning algorithm. Starting from this analysis, we briefly review proposals for discovering optimal solutions and suggest conditions for designing architectures tailored to a given task.
---
paper_title: A technique for the selection of kernel-function parameters in RBF neural networks for classification of remote-sensing images

paper_content:

A supervised technique for training radial basis function (RBF) neural network classifiers is proposed. Such a technique, unlike traditional ones, considers the class memberships of training samples to select the centers and widths of the kernel functions associated with the hidden neurons of an RBF network. The result is twofold: a significant reduction in the overall classification error made by the classifier and a more stable behavior of the classification error versus variations in both the number of hidden units and the initial parameters of the training process.
---
paper_title: Feature extraction and pattern classification of remote sensing data by a modular neural system

paper_content:

A modular neural network architecture has been used for the classification of remote sensed data in two experiments carried out to study two different but rather usual situations in real remote sensing applications. Such situations concern the availability of high-dimensional data in the first setting and an imperfect data set with a limited number of features in the second. The learning task of the supervised multilayer perceptron classifier has been made more efficient by preprocessing the input with unsupervised neural modules for feature discovery. The linear propagation network is introduced in the first experiment to evaluate the effectiveness of the neural data compression stage before classification, whereas in the second experiment data clustering before labeling is evaluated by the Kohonen self-organizing feature map network. The results of the two experiments confirm that modular learning performs better than nonmodular learning with respect to both learning quality and speed. © 1996 Society of Photo-Optical Instrumentation Engineers. Subject terms: neural network; remote sensing; classification; supervised and un- supervised learning.
---
paper_title: Optimal learning in artificial neural networks: A review of theoretical results

paper_content:

Abstract The effectiveness of connectionist models in emulating intelligent behaviour and solving significant practical problems is strictly related to the capability of the learning algorithms to find optimal or near-optimal solutions and to generalize to new examples. This paper reviews some theoretical contributions to optimal learning in the attempt to provide a unified view and give the state of the art in the field. The focus of the review is on the problem of local minima in the cost function that is likely to affect more or less any learning algorithm. Starting from this analysis, we briefly review proposals for discovering optimal solutions and suggest conditions for designing architectures tailored to a given task.
---
paper_title: Complex Systems and Cognitive Processes

paper_content:

This book shows that the science of complex systems, which stresses the importance of self-organizing processes, can make a decisive contribution to solving many problems in artificial intelligence. Artificial cognitive systems are important in view of their potential applications, and it can be expected that their study will shed light on biological cognitive systems. The new "neurally inspired" information science proposed in this book is fast becoming a promising workshop for the construction of models capable of emulating cognitive behaviour. After a general introduction to the theory of complex systems, the book gives a thorough treatment of neural networks, which are the most successful and the most thoroughly studied dynamical cognitive systems. Attention is also devoted to other classes of artificial cognitive systems, in particular to classifier systems, which provide an important link between the dynamical and the inferential approach to artificial intelligence. The book can be used as a textbook, since it does not require previous knowledge of the topic, and should also be interesting for researchers in this field, since it links formerly separate lines of research.
---
paper_title: Learning without Local Minima in Radial Basis Function Networks

paper_content:

Learning from examples plays a central role in artificial neural networks. The success of many learning schemes is not guaranteed, however, since algorithms like backpropagation may get stuck in local minima, thus providing suboptimal solutions. For feedforward networks, optimal learning can be achieved provided that certain conditions on the network and the learning environment are met. This principle is investigated for the case of networks using radial basis functions (RBF). It is assumed that the patterns of the learning environment are separable by hyperspheres. In that case, we prove that the attached cost function is local minima free with respect to all the weights. This provides us with some theoretical foundations for a massive application of RBF in pattern recognition. >
---
paper_title: The self-organizing map

paper_content:

The self-organized map, an architecture suggested for artificial neural networks, is explained by presenting simulation experiments and practical applications. The self-organizing map has the property of effectively creating spatially organized internal representations of various features of input signals and their abstractions. One result of this is that the self-organization process can discover semantic relationships in sentences. Brain maps, semantic maps, and early work on competitive learning are reviewed. The self-organizing map algorithm (an algorithm which order responses spatially) is reviewed, focusing on best matching cell selection and adaptation of the weight vectors. Suggestions for applying the self-organizing map algorithm, demonstrations of the ordering process, and an example of hierarchical clustering of data are presented. Fine tuning the map by learning vector quantization is addressed. The use of self-organized maps in practical speech recognition and a simulation experiment on semantic mapping are discussed. >
---
paper_title: The Design and Evolution of Modular Neural Network Architectures

paper_content:

To investigate the relations between structure and function in both artificial and natural neural networks, we present a series of simulations and analyses with modular neural networks. We suggest a number of design principles in the form of explicit ways in which neural modules can cooperate in recognition tasks. These results may supplement recent accounts of the relation between structure and function in the brain. The networks used consist of several modules, standard subnetworks that serve as higher order units with a distinct structure and function. The simulations rely on a particular network module called the categorizing and learning module. This module, developed mainly for unsupervised categorization and learning, is able to adjust its local learning dynamics. The way in which modules are interconnected is an important determinant of the learning and categorization behaviour of the network as a whole. Based on arguments derived from neuroscience, psychology, computational learning theory, and hardware implementation, a framework for the design of such modular networks is presented. A number of small-scale simulation studies shows how intermodule connectivity patterns implement ''neural assemblies'' that induce a particular category structure in the network. Learning and categorization improves because the induced categories are more compatible with the structure of the task domain. In addition to structural compatibility, two other principles of design are proposed that underlie information processing in interactive activation networks: replication and recurrence. Because a general theory for relating network architectures to specific neural functions does not exist, we extend the biological metaphor of neural networks, by applying genetic algorithms (a biocomputing method for search and optimization based on natural selection and evolution) to search for optimal modular network architectures for learning a visual categorization task. The best performing network architectures seemed to have reproduced some of the overall characteristics of the natural visual system, such as the organization of coarse and fine processing of stimuli in separate pathways. A potentially important result is that a genetically defined initial architecture cannot only enhance learning and recognition performance, but it can also induce a system to better generalize its learned behaviour to instances never encountered before. This may explain why for many vital learning tasks in organisms only a minimal exposure to relevant stimuli is necessary.
---
paper_title: A Growing Neural Gas Network Learns Topologies

paper_content:

An incremental network model is introduced which is able to learn the important topological relations in a given set of input vectors by means of a simple Hebb-like learning rule. In contrast to previous approaches like the "neural gas" method of Martinetz and Schulten (1991, 1994), this model has no parameters which change over time and is able to continue learning, adding units and connections, until a performance criterion has been met. Applications of the model include vector quantization, clustering, and interpolation.
---
paper_title: Fuzzy Kohonen clustering networks

paper_content:

The authors propose a fuzzy Kohonen clustering network which integrates the fuzzy c-means (FCM) model into the learning rate and updating strategies of the Kohonen network. This yields an optimization problem related to FCM, and the numerical results show improved convergence as well as reduced labeling errors. It is proved that the proposed scheme is equivalent to the c-means algorithms. The new method can be viewed as a Kohonen type of FCM, but it is self-organizing, since the size of the update neighborhood and the learning rate in the competitive layer are automatically adjusted during learning. Anderson's IRIS data were used to illustrate this method. The results are compared with the standard Kohonen approach. >
---
paper_title: Topology representing networks

paper_content:

Abstract A Hebbian adaptation rule with winner-take-all like competition is introduced. It is shown that this competitive Hebbian rule forms so-called Delaunay triangulations, which play an important role in computational geometry for efficiently solving proximity problems. Given a set of neural units i, i = 1,…, N, the synaptic weights of which can be interpreted as pointers wi, i = 1,…, N in RD, the competitive Hebbian rule leads to a connectivity structure between the units i that corresponds to the Delaunay triangulation of the set of pointers wi. Such competitive Hebbian rule develops connections (Cij > 0) between neural units i, j with neighboring receptive fields (Voronoi polygons) Vi, Vj, whereas between all other units i, j no connections evolve (Cij = 0). Combined with a procedure that distributes the pointers wi over a given feature manifold M, for example, a submanifold M ⊂ RD, the competitive Hebbian rule provides a novel approach to the problem of constructing topology preserving feature maps and representing intricately structured manifolds. The competitive Hebbian rule connects only neural units, the receptive fields (Voronoi polygons) Vi, Vj of which are adjacent on the given manifold M. This leads to a connectivity structure that defines a perfectly topology preserving map and forms a discrete, path preserving representation of M, also in cases where M has an intricate topology. This makes this novel approach particularly useful in all applications where neighborhood relations have to be exploited or the shape and topology of submanifolds have to be take into account.
---
paper_title: Fuzzy Kohonen clustering networks

paper_content:

The authors propose a fuzzy Kohonen clustering network which integrates the fuzzy c-means (FCM) model into the learning rate and updating strategies of the Kohonen network. This yields an optimization problem related to FCM, and the numerical results show improved convergence as well as reduced labeling errors. It is proved that the proposed scheme is equivalent to the c-means algorithms. The new method can be viewed as a Kohonen type of FCM, but it is self-organizing, since the size of the update neighborhood and the learning rate in the competitive layer are automatically adjusted during learning. Anderson's IRIS data were used to illustrate this method. The results are compared with the standard Kohonen approach. >
---
paper_title: Neural networks and physical systems with emergent collective computational abilities

paper_content:

Abstract ::: Computational properties of use of biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.
---
paper_title: A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recognition Machine

paper_content:

A neural network architecture for the learning of recognition categories is derived. Real-time network dynamics are completely characterized through mathematical analysis and computer simulations. The architecture self-organizes and self-stabilizes its recognition codes in response to arbitrary orderings of arbitrarily many and arbitrarily complex binary input patterns. Top-down attentional and matching mechanisms are critical in self-stabilizing the code learning process. The architecture embodies a parallel search scheme which updates itself adaptively as the learning process unfolds. After learning self-stabilizes, the search process is automatically disengaged. Thereafter input patterns directly access their recognition codes without any search. Thus recognition time does not grow as a function of code complexity. A novel input pattern can directly access a category if it shares invariant properties with the set of familiar exemplars of that category. These invariant properties emerge in the form of learned critical feature patterns, or prototypes. The architecture possesses a context-sensitive self-scaling property which enables its emergent critical feature patterns to form. They detect and remember statistically predictive configurations of featural elements which are derived from the set of all input patterns that are ever experienced. Four types of attentional process—priming, gain control, vigilance, and intermodal competition—are mechanistically characterized. Top—down priming and gain control are needed for code matching and self-stabilization. Attentional vigilance determines how fine the learned categories will be. If vigilance increases due to an environmental disconfirmation, then the system automatically searches for and learns finer recognition categories. A new nonlinear matching law (the ⅔ Rule) and new nonlinear associative laws (the Weber Law Rule, the Associative Decay Rule, and the Template Learning Rule) are needed to achieve these properties. All the rules describe emergent properties of parallel network interactions. The architecture circumvents the noise, saturation, capacity, orthogonality, and linear predictability constraints that limit the codes which can be stably learned by alternative recognition models.
---
paper_title: Complex Systems and Cognitive Processes

paper_content:

This book shows that the science of complex systems, which stresses the importance of self-organizing processes, can make a decisive contribution to solving many problems in artificial intelligence. Artificial cognitive systems are important in view of their potential applications, and it can be expected that their study will shed light on biological cognitive systems. The new "neurally inspired" information science proposed in this book is fast becoming a promising workshop for the construction of models capable of emulating cognitive behaviour. After a general introduction to the theory of complex systems, the book gives a thorough treatment of neural networks, which are the most successful and the most thoroughly studied dynamical cognitive systems. Attention is also devoted to other classes of artificial cognitive systems, in particular to classifier systems, which provide an important link between the dynamical and the inferential approach to artificial intelligence. The book can be used as a textbook, since it does not require previous knowledge of the topic, and should also be interesting for researchers in this field, since it links formerly separate lines of research.
---
paper_title: The self-organizing map

paper_content:

The self-organized map, an architecture suggested for artificial neural networks, is explained by presenting simulation experiments and practical applications. The self-organizing map has the property of effectively creating spatially organized internal representations of various features of input signals and their abstractions. One result of this is that the self-organization process can discover semantic relationships in sentences. Brain maps, semantic maps, and early work on competitive learning are reviewed. The self-organizing map algorithm (an algorithm which order responses spatially) is reviewed, focusing on best matching cell selection and adaptation of the weight vectors. Suggestions for applying the self-organizing map algorithm, demonstrations of the ordering process, and an example of hierarchical clustering of data are presented. Fine tuning the map by learning vector quantization is addressed. The use of self-organized maps in practical speech recognition and a simulation experiment on semantic mapping are discussed. >
---
paper_title: Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps

paper_content:

A neural network architecture is introduced for incremental supervised learning of recognition categories and multidimensional maps in response to arbitrary sequences of analog or binary input vectors, which may represent fuzzy or crisp sets of features. The architecture, called fuzzy ARTMAP, achieves a synthesis of fuzzy logic and adaptive resonance theory (ART) neural networks by exploiting a close formal similarity between the computations of fuzzy subsethood and ART category choice, resonance, and learning. Four classes of simulation illustrated fuzzy ARTMAP performance in relation to benchmark backpropagation and generic algorithm systems. These simulations include finding points inside versus outside a circle, learning to tell two spirals apart, incremental approximation of a piecewise-continuous function, and a letter recognition database. The fuzzy ARTMAP system is also compared with Salzberg's NGE systems and with Simpson's FMMC system. >
---
paper_title: Art 2: Self-Organization Of Stable Category Recognition Codes For Analog Input Patterns

paper_content:

Adaptive resonance architectures are neural networks that self-organize stable pattern recognition codes in real-time in response to arbitrary sequences of input patterns. This article introduces ART 2, a class of adaptive resonance architectures which rapidly self-organize pattern recognition categories in response to arbitrary sequences of either analog of binary input patterns. In order to cope with arbitrary sequences of analog input patterns, ART 2 architectures embody solutions to a number of design principles, such as the stability-plasticity tradeoff, the search-direct access tradeoff, and the match-reset tradeoff. In these architectures, top-down learned expectation and matching mechanisms are critical in self-stabilizing the code learning process. A parallel search scheme updates itself adaptively as the learning process unfolds, and realizes a form of real-time hypothesis discovery, testing, learning, and recognition. After learning self-stabilizes, the search process is automatically disengaged. Thereafter input patterns directly access their recognition codes without any search. Thus recognition time for familiar inputs does not increase with the complexity of the learned code. A novel input pattern can directly access a category if it shares invariant properties with the set of familiar exemplars of that category. A parameter called the attentional vigilance parameter determines how fine the categories will be. If vigilance increases (decreases) due to environmental feedback, then the system automatically searches for and learns finer (coarser) recognition categories. Gain control parameters enable the architecture to suppress noise up to a prescribed level. The architecture's global design enables it to learn effectively despite the high degree of nonlinearity of such mechanisms.
---
paper_title: Signal and Image Processing with Neural Networks: A C++ Sourcebook

paper_content:

The Role of Neural Networks in Signal and Image Processing Neurons in the Complex Domain Data Preparation for Neural Networks Frequency-Domain Techniques Time/Frequency Localization Time/Frequency Applications Image Processing in the Frequency Domain Moment-Based Image Features Tone/Texture Descriptors Using the MLFN Program Appendix Bibliography Index.
---
paper_title: Maximum likelihood from incomplete data via the EM algorithm

paper_content:

Vibratory power unit for vibrating conveyers and screens comprising an asynchronous polyphase motor, at least one pair of associated unbalanced masses disposed on the shaft of said motor, with the first mass of a pair of said unbalanced masses being rigidly fastened to said shaft and with said second mass of said pair being movably arranged relative to said first mass, means for controlling and regulating the conveying rate during conveyer operation by varying the rotational speed of said motor between predetermined minimum and maximum values, said second mass being movably outwardly by centrifugal force against the pressure of spring means, said spring means being prestressed in such a manner that said second mass is, at rotational motor speeds lower than said minimum speed, held in its initial position, and at motor speeds between said lower and upper values in positions which are radially offset with respect to the axis of said motor to an extent depending on the value of said rotational motor speed.
---
paper_title: Novel neural network model combining radial basis function, competitive Hebbian learning rule, and fuzzy simplified adaptive resonance theory

paper_content:

In the first part of this paper a new on-line fully self- organizing artificial neural network model (FSONN), pursuing dynamic generation and removal of neurons and synaptic links, is proposed. The model combines properties of the self- organizing map (SOM), fuzzy c-means (FCM), growing neural gas (GNG) and fuzzy simplified adaptive resonance theory (Fuzzy SART) algorithms. In the second part of the paper experimental results are provided and discussed. Our conclusion is that the proposed connectionist model features several interesting properties, such as the following: (1) the system requires no a priori knowledge of the dimension, size and/or adjacency structure of the network; (2) with respect to other connectionist models found in the literature, the system can be employed successfully in: (a) a vector quantization; (b) density function estimation; and (c) structure detection in input data to be mapped topologically correctly onto an output lattice pursuing dimensionality reduction; and (3) the system is computationally efficient, its processing time increasing linearly with the number of neurons and synaptic links.
---
paper_title: ;Neural-gas' network for vector quantization and its application to time-series prediction.

paper_content:

A neural network algorithm based on a soft-max adaptation rule is presented. This algorithm exhibits good performance in reaching the optimum minimization of a cost function for vector quantization data compression. The soft-max rule employed is an extension of the standard K-means clustering procedure and takes into account a neighborhood ranking of the reference (weight) vectors. It is shown that the dynamics of the reference (weight) vectors during the input-driven adaptation procedure are determined by the gradient of an energy function whose shape can be modulated through a neighborhood determining parameter and resemble the dynamics of Brownian particles moving in a potential determined by the data point density. The network is used to represent the attractor of the Mackey-Glass equation and to predict the Mackey-Glass time series, with additional local linear mappings for generating output values. The results obtained for the time-series prediction compare favorably with the results achieved by backpropagation and radial basis function networks.
---
paper_title: A Growing Neural Gas Network Learns Topologies

paper_content:

An incremental network model is introduced which is able to learn the important topological relations in a given set of input vectors by means of a simple Hebb-like learning rule. In contrast to previous approaches like the "neural gas" method of Martinetz and Schulten (1991, 1994), this model has no parameters which change over time and is able to continue learning, adding units and connections, until a performance criterion has been met. Applications of the model include vector quantization, clustering, and interpolation.
---
paper_title: Fuzzy Kohonen clustering networks

paper_content:

The authors propose a fuzzy Kohonen clustering network which integrates the fuzzy c-means (FCM) model into the learning rate and updating strategies of the Kohonen network. This yields an optimization problem related to FCM, and the numerical results show improved convergence as well as reduced labeling errors. It is proved that the proposed scheme is equivalent to the c-means algorithms. The new method can be viewed as a Kohonen type of FCM, but it is self-organizing, since the size of the update neighborhood and the learning rate in the competitive layer are automatically adjusted during learning. Anderson's IRIS data were used to illustrate this method. The results are compared with the standard Kohonen approach. >
---

Title: A Survey of Fuzzy Clustering Algorithms for Pattern Recognition - Part 11

Section 1: FUZZY MEMBERSHIP AND PROBABILITY DENSITY FUNCTIONS
Description 1: This section proposes a brief review of probabilistic and possibilistic fuzzy membership concepts, to be compared with Bayes' view of posterior probability and likelihood.

Section 2: ON-LINE VERSUS OFF-LINE MODEL ADAPTATION
Description 2: This section discusses the principles of batch (off-line) and on-line learning methods, their advantages, and issues like real-time response, noise sensitivity, and parameter updates.

Section 3: PROTOTYPE VECTOR EDITING SCHEME
Description 3: This section covers the clustering algorithms employing two main schemes for reference vector generation, namely clustering-by-selection and clustering-by-replacement.

Section 4: BEYOND ERROR GRADIENT DESCENT: ADVANCED TECHNIQUES FOR LEARNING FROM DATA
Description 4: This section presents advanced learning techniques developed to deal more efficiently with local minima and generalization issues than traditional gradient descent methods.

Section 5: TOPOLOGICALLY CORRECT MAPPING
Description 5: This section explains topologically correct mapping and the relevant methodologies like the competitive Hebbian rule to generate synaptic connections in self-organizing neural network models.

Section 6: ARTIFICIAL COGNITIVE SYSTEMS AND ECOLOGICAL NETS
Description 6: This section discusses the requirements for artificial cognitive systems and the need to model neural networks alongside their environments, evolving towards ecological networks.

Section 7: ON FUZZY CLUSTERING ALGORITHMS
Description 7: This section reviews the types and principles of fuzzy clustering algorithms, highlighting soft competitive learning as core to the fuzzification of clustering schemes.

Section 8: CONCLUSION
Description 8: This section summarizes the paper's interpretation of fuzzy clustering as synonymous with soft competitive parameter adaptation and outlines the key features compared in the survey.