Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies
---
paper_title: Multi-criteria reinforcement learning

paper_content:

Ve cOllf:iider multi- criteria f:iequent,ial decision making problems where the vcctor-"valucd evaluations arc compared by a given, fixed total ordering. Condit.ions for the opt.irnality of statiOIl<-l,r}' p()lichs ;-weI the Bellman opti­ malit,y equatio n a re given for a. speci al, but. important class of problems ''v hell the eval­ uation of policies can be computed for the criteria, independently of each other. The anal)'sis requires special cafC as t.he t.opol­ ag,Y int.roduced by polnL\visc convergence a.ncl the or<1cr- topology introduced by the prefer­ ence order arc in general incompatible. Re­ inf orcement. learning algorithms are proposed and analY7,ed.
---
paper_title: INTRODUCTION TO MACHINE LEARNINGIntroduction to Machine Learning

paper_content:

This chapter introduces the foundations of machine learning that are necessary to motivate its combination to evolutionary algorithms and image processing. In the chapter, we will describe several machine learning tools, suitable for a wide variety of data and tasks. The goal is not to describe the total number of techniques, but instead to present a general view of the field through the description of the mail problems considered in machine learning.
---
paper_title: Generating fc fuzzy rule systems from data using evolution strategies

paper_content:

Sophisticated fuzzy rule systems are supposed to be flexible, complete, consistent and compact (FC/sup 3/). Flexibility, and consistency are essential for fuzzy systems to exhibit an excellent performance and to have a clear physical meaning, while compactness is crucial when the number of the input variables increases. However, the completeness and consistency conditions are often violated if a fuzzy system is generated from data collected from real world applications. A systematic design paradigm is proposed using evolution strategies. The structure of the fuzzy rules, which determines the compactness of the fuzzy systems, is evolved along with the parameters of the fuzzy systems. Special attention has been paid to the completeness and consistency of the rule base. The completeness is guaranteed by checking the completeness of the fuzzy partitioning of input variables and the completeness of the rule structure. An index of inconsistency is suggested with the help of a fuzzy similarity which can prevent the algorithm from generating rules that seriously contradict with each other or with the heuristic knowledge. In addition, soft T-norm and BADD defuzzification are introduced and optimized to increase the flexibility of the fuzzy system. The proposed approach is applied to the design of a distance controller for cars. It is verified that a FC/sup 3/ fuzzy system works very well both, for training and test driving situations, especially when the training data are insufficient.
---
paper_title: Model-based Gaussian and non-Gaussian Clustering

paper_content:

Abstract : The classification maximum likelihood approach is sufficiently general to encompass many current clustering algorithms, including those based on the sum of squares criterion and on the criterion of Friedman and Rubin (1967). However, as currently implemented, it does not allow the specification of which features (orientation, size and shape) are to be common to all clusters and which may differ between clusters. Also, it is restricted to Gaussian distributions and it does not allow for noise. We propose ways of overcoming these limitations. A reparameterization of the covariance matrix allows us to specify that some features, but not all, be the same for all clusters. A practical framework for non-Gaussian clustering is outlined, and a means of incorporating noise in the form of a Poisson process is described. An approximate Bayesian method for choosing the number of clusters is given. The performance of the proposed methods is studied by simulation, with encouraging results. The methods are applied to the analysis of a data set arising in the study of diabetes, and the results seem better than those of previous analyses. (RH)
---
paper_title: Support-vector networks

paper_content:

Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
---
paper_title: A closer look at drawbacks of minimizing weighted sums of objectives for Pareto set generation in multicriteria optimization problems

paper_content:

A standard technique for generating the Pareto set in multicriteria optimization problems is to minimize (convex) weighted sums of the different objectives for various different settings of the weights. However, it is well-known that this method succeeds in getting points from all parts of the Pareto set only when the Pareto curve is convex. This article provides a geometrical argument as to why this is the case.
---
paper_title: Improving generalization of MLPs with multi-objective optimization

paper_content:

Abstract This paper presents a new learning scheme for improving generalization of multilayer perceptrons. The algorithm uses a multi-objective optimization approach to balance between the error of the training data and the norm of network weight vectors to avoid overfitting. The results are compared with support vector machines and standard backpropagation.
---
paper_title: ROC Graphs : Notes and Practical Considerations for Data Mining Researchers

paper_content:

Receiver Operating Characteristics (ROC) graphs are a useful technique for organizing classifiers and visualizing their performance. ROC graphs are commonly used in medical decision making, and in recent years have been increasingly adopted in the machine learning and data mining research communities. Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice. This article serves both as a tutorial introduction to ROC graphs and as a practical guide for using them in research.
---
paper_title: Fuzzy Modeling of High-Dimensional Systems: Complexity Reduction and Interpretability Improvement

paper_content:

Fuzzy modeling of high-dimensional systems is a challenging topic. This paper proposes an effective approach to data-based fuzzy modeling of high-dimensional systems. An initial fuzzy rule system is generated based on the conclusion that optimal fuzzy rules cover extrema. Redundant rules are removed based on a fuzzy similarity measure. Then, the structure and parameters of the fuzzy system are optimized using a genetic algorithm and the gradient method. During optimization, rules that have a very low firing strength are deleted. Finally, interpretability of the fuzzy system is improved by fine training the fuzzy rules with regularization. The resulting fuzzy system generated by this method has the following distinct features: (1) the fuzzy system is quite simplified; (2) the fuzzy system is interpretable; and (3) the dependencies between the inputs and the output are clearly shown. This method has successfully been applied to a system that has 11 inputs and one output with 20 000 training data and 80 000 test data.
---
paper_title: Sparse coding with an overcomplete basis set: A strategy employed by V1?

paper_content:

The spatial receptive fields of simple cells in mammalian striate cortex have been reasonably well described physiologically and can be characterized as being localized, oriented, and ban@ass, comparable with the basis functions of wavelet transforms. Previously, we have shown that these receptive field properties may be accounted for in terms of a strategy for producing a sparse distribution of output activity in response to natural images. Here, in addition to describing this work in a more expansive fashion, we examine the neurobiological implications of sparse coding. Of particular interest is the case when the code is overcomplete--i.e., when the number of code elements is greater than the effective dimensionality of the input space. Because the basis functions are non-orthogonal and not linearly independent of each other, sparsifying the code will recruit only those basis functions necessary for representing a given input, and so the input-output function will deviate from being purely linear. These deviations from linearity provide a potential explanation for the weak forms of non-linearity observed in the response properties of cortical simple cells, and they further make predictions about the expected interactions among units in response to naturalistic stimuli. © 1997 Elsevier Science Ltd
---
paper_title: Ensemble Learning via Negative Correlation

paper_content:

This paper presents a learning approach, i.e. negative correlation learning, for neural network ensembles. Unlike previous learning approaches for neural network ensembles, negative correlation learning attempts to train individual networks in an ensemble and combines them in the same learning process. In negative correlation learning, all the individual networks in the ensemble are trained simultaneously and interactively through the correlation penalty terms in their error functions. Rather than producing unbiased individual networks whose errors are uncorrelated, negative correlation learning can create negatively correlated networks to encourage specialisation and cooperation among the individual networks. Empirical studies have been carried out to show why and how negative correlation learning works. The experimental results show that negative correlation learning can produce neural network ensembles with good generalisation ability.
---
paper_title: Multiobjective data clustering

paper_content:

Conventional clustering algorithms utilize a single criterion that may not conform to the diverse shapes of the underlying clusters. We offer a new clustering approach that uses multiple clustering objective functions simultaneously. The proposed multiobjective clustering is a two-step process. It includes detection of clusters by a set of candidate objective functions as well as their integration into the target partition. A key ingredient of the approach is a cluster goodness junction that evaluates the utility of multiple clusters using re-sampling techniques. Multiobjective data clustering is obtained as a solution to a discrete optimization problem in the space of clusters. At meta-level, our algorithm incorporates conflict resolution techniques along with the natural data constraints. An empirical study on a number of artificial and real-world data sets demonstrates that multiobjective data clustering leads to valid and robust data partitions.
---
paper_title: Dynamic Weighted Aggregation for evolutionary multi-objective optimization: why does it work and how?

paper_content:

Evolutionary Dynamic Weighted Aggregation (EDWA) has shown to be both effective and computationally efficient [1] for multi-objective optimization (MOO). Besides, it was also found empirically and surprisingly that EDWA was able to deal with multi-objective optimization problems with a concave Pareto front, which has proved to be beyond the capability of the Conventional Weighted Aggregation (CWA) methods [2]. In this paper, a theory on why CWA fails for multi-objective problems with a concave Pareto front is provided schematically. According to this theory, it can easily be explained why EDWA has worked well for both convex and concave multi-objective problems. Simulation examples are conducted on various test functions to support our theory. It is concluded that EDWA is an effective and efficient method for solving multi-objective optimization problems.
---
paper_title: Evolutionary Multi-objective Optimization for Simultaneous Generation of Signal-type and Symbol-type Representations

paper_content:

It has been a controversial issue in the research of cognitive science and artificial intelligence whether signal-type representations (typically connectionist networks) or symbol-type representations (e.g., semantic networks, production systems) should be used. Meanwhile, it has also been recognized that both types of information representations might exist in the human brain. In addition, symbol-type representations are often very helpful in gaining insights into unknown systems. For these reasons, comprehensible symbolic rules need to be extracted from trained neural networks. In this paper, an evolutionary multi-objective algorithm is employed to generate multiple models that facilitate the generation of signal-type and symbol-type representations simultaneously. It is argued that one main difference between signal-type and symbol-type representations lies in the fact that the signal-type representations are models of a higher complexity (fine representation), whereas symbol-type representations are models of a lower complexity (coarse representation). Thus, by generating models with a spectrum of model complexity, we are able to obtain a population of models of both signal-type and symbol-type quality, although certain post-processing is needed to get a fully symbol-type representation. An illustrative example is given on generating neural networks for the breast cancer diagnosis benchmark problem.
---
paper_title: INTRODUCTION TO MACHINE LEARNINGIntroduction to Machine Learning

paper_content:

This chapter introduces the foundations of machine learning that are necessary to motivate its combination to evolutionary algorithms and image processing. In the chapter, we will describe several machine learning tools, suitable for a wide variety of data and tasks. The goal is not to describe the total number of techniques, but instead to present a general view of the field through the description of the mail problems considered in machine learning.
---
paper_title: Multi-objective machine learning

paper_content:

Multi-Objective Clustering, Feature Extraction and Feature Selection.- Feature Selection Using Rough Sets.- Multi-Objective Clustering and Cluster Validation.- Feature Selection for Ensembles Using the Multi-Objective Optimization Approach.- Feature Extraction Using Multi-Objective Genetic Programming.- Multi-Objective Learning for Accuracy Improvement.- Regression Error Characteristic Optimisation of Non-Linear Models.- Regularization for Parameter Identification Using Multi-Objective Optimization.- Multi-Objective Algorithms for Neural Networks Learning.- Generating Support Vector Machines Using Multi-Objective Optimization and Goal Programming.- Multi-Objective Optimization of Support Vector Machines.- Multi-Objective Evolutionary Algorithm for Radial Basis Function Neural Network Design.- Minimizing Structural Risk on Decision Tree Classification.- Multi-objective Learning Classifier Systems.- Multi-Objective Learning for Interpretability Improvement.- Simultaneous Generation of Accurate and Interpretable Neural Network Classifiers.- GA-Based Pareto Optimization for Rule Extraction from Neural Networks.- Agent Based Multi-Objective Approach to Generating Interpretable Fuzzy Systems.- Multi-objective Evolutionary Algorithm for Temporal Linguistic Rule Extraction.- Multiple Objective Learning for Constructing Interpretable Takagi-Sugeno Fuzzy Model.- Multi-Objective Ensemble Generation.- Pareto-Optimal Approaches to Neuro-Ensemble Learning.- Trade-Off Between Diversity and Accuracy in Ensemble Generation.- Cooperative Coevolution of Neural Networks and Ensembles of Neural Networks.- Multi-Objective Structure Selection for RBF Networks and Its Application to Nonlinear System Identification.- Fuzzy Ensemble Design through Multi-Objective Fuzzy Rule Selection.- Applications of Multi-Objective Machine Learning.- Multi-Objective Optimisation for Receiver Operating Characteristic Analysis.- Multi-Objective Design of Neuro-Fuzzy Controllers for Robot Behavior Coordination.- Fuzzy Tuning for the Docking Maneuver Controller of an Automated Guided Vehicle.- A Multi-Objective Genetic Algorithm for Learning Linguistic Persistent Queries in Text Retrieval Environments.- Multi-Objective Neural Network Optimization for Visual Object Detection.
---
paper_title: Learning with multi-objective criteria

paper_content:

The paper presents a new algorithm for learning with neural networks based on multi objective performance criteria. It considers three performance indices (or cost functions) as the objectives, which are the Euclidean distance and maximum difference measurements between the real nonlinear system and the nonlinear model (L/sub 2/, L/sub /spl infin// norms), and the complexity measure of the nonlinear model, instead of a single performance index. An algorithm based on the method of inequalities, least squares and genetic algorithms is developed for optimising over the multi objective criteria. Genetic algorithms are also used simultaneously for model selection in which the structure of the neural networks are determined. The Volterra polynomial basis function network and the Gaussian radial basis function network are applied to the identification of a liquid level nonlinear system.
---
paper_title: Harmonic competition: a self-organizing multiple criteria optimization

paper_content:

Harmonic competition is a learning strategy based upon winner-take-all or winner-take-quota with respect to a composite of heterogeneous subcosts. This learning is unsupervised and organizes itself. The subcosts may conflict with each other. Thus, the total learning system realizes a self-organizing multiple criteria optimization. The subcosts are combined additively and multiplicatively using adjusting parameters. For such a total cost, a general successive learning algorithm is derived first. Then, specific problems in the Euclidian space are addressed. Vector quantization with various constraints and traveling salesperson problems are selected as test problems. The former is a typical class of problems where the number of neurons is less than that of the data. The latter is an opposite case. Duality exists in these two classes. In both cases, the combination parameters of the subcosts show wide dynamic ranges in the course of learning. It is possible, however, to decide the parameter control from the structure of the total cost. This method finds a preferred solution from the Pareto optimal set of the multiple object optimization. Controlled mutations motivated by genetic algorithms are proved to be effective in finding near-optimal solutions.
---
paper_title: Evolving optimal feature extraction using multi-objective genetic programming: a methodology and preliminary study on edge detection

paper_content:

In this paper we describe a generic methodology to create an "optimal" feature extraction pre-processing stage for pattern classification. Our aim is to map the input data into a new, one-dimensional feature space in which separability is maximized under a simple thresholding classification. We have used multi-objective genetic programming with Pareto strength-based ranking to bias the selection procedure. The methodology is applied to the edge detection problem in image processing; we make quantitative comparison with the pre-processing stages of the well-known Canny edge detector using synthetic and real-world edge data and conclude that the performance of our evolutionary-based method is much superior to the Canny algorithm based on the criterion of minimum Bayes risk.
---
paper_title: A Multi-Objective Genetic Algorithm for Learning Linguistic Persistent Queries in Text Retrieval Environments

paper_content:

Summary. Persistent queries are a specific kind of queries used in information retrieval systems to represent a user’s long-term standing information need. These queries can present many different structures, being the “bag of words” that most commonly used. They can be sometimes formulated by the user, although this task is usually difficult for him and the persistent query is then automatically derived from a set of sample documents he provides. In this work we aim at getting persistent queries with a more representative structure for text retrieval issues. To do so, we make use of soft computing tools: linguistic information is considered for weighting the terms of Boolean queries by means of ordinal linguistic values (linguistic queries), and multiobjective evolutionary algorithms are applied to build the linguistic persistent query. Experimental results will show how using an expressive linguistic information-based query structure and a proper learning process to derive it, we can get more flexible, comprehensible and expressive user profiles.
---
paper_title: Pareto evolutionary neural networks

paper_content:

For the purposes of forecasting (or classification) tasks neural networks (NNs) are typically trained with respect to Euclidean distance minimization. This is commonly the case irrespective of any other end user preferences. In a number of situations, most notably time series forecasting, users may have other objectives in addition to Euclidean distance minimization. Recent studies in the NN domain have confronted this problem by propagating a linear sum of errors. However this approach implicitly assumes a priori knowledge of the error surface defined by the problem, which, typically, is not the case. This study constructs a novel methodology for implementing multiobjective optimization within the evolutionary neural network (ENN) domain. This methodology enables the parallel evolution of a population of ENN models which exhibit estimated Pareto optimality with respect to multiple error measures. A new method is derived from this framework, the Pareto evolutionary neural network (Pareto-ENN). The Pareto-ENN evolves a population of models that may be heterogeneous in their topologies inputs and degree of connectivity, and maintains a set of the Pareto optimal ENNs that it discovers. New generalization methods to deal with the unique properties of multiobjective error minimization that are not apparent in the uni-objective case are presented and compared on synthetic data, with a novel method based on bootstrapping of the training data shown to significantly improve generalization ability. Finally experimental evidence is presented in this study demonstrating the general application potential of the framework by generating populations of ENNs for forecasting 37 different international stock indexes.
---
paper_title: Improving generalization of MLPs with multi-objective optimization

paper_content:

Abstract This paper presents a new learning scheme for improving generalization of multilayer perceptrons. The algorithm uses a multi-objective optimization approach to balance between the error of the training data and the norm of network weight vectors to avoid overfitting. The results are compared with support vector machines and standard backpropagation.
---
paper_title: Structural risk minimization on decision trees using an evolutionary multiobjective optimization

paper_content:

Inducing decision trees is a popular method in machine learning. The information gain computed for each attribute and its threshold helps finding a small number of rules for data classification. However, there has been little research on how many rules are appropriate for a given set of data. In this paper, an evolutionary multi-objective optimization approach with genetic programming will be applied to the data classification problem in order to find the minimum error rate for each size of decision trees. Following structural risk minimization suggested by Vapnik, we can determine a desirable number of rules with the best generalization performance. A hierarchy of decision trees for classification performance can be provided and it is compared with C4.5 application.
---
paper_title: Multi-Objective Programming in SVMs

paper_content:

We propose a general framework for support vector machines (SVM) based on the principle of multi-objective optimization. The learning of SVMs is formulated as a multi-objective program by setting two competing goals to minimize the empirical risk and minimize the model capacity. Distinct approaches to solving the MOP introduce various SVM formulations. The proposed framework enables a more effective minimization of the VC bound on the generalization risk. We develop a feature selection approach based on the MOP framework and demonstrate its effectiveness on hand-written digit data.
---
paper_title: Neural network regularization and ensembling using multi-objective evolutionary algorithms

paper_content:

Regularization is an essential technique to improve generalization of neural networks. Traditionally, regularization is conducted by including an additional term in the cost function of a learning algorithm. One main drawback of these regularization techniques is that a hyperparameter that determines to which extension the regularization influences the learning algorithm must be determined beforehand. This paper addresses the neural network regularization problem from a multi-objective optimization point of view. During the optimization, both structure and parameters of the neural network will be optimized. A slightly modified version of two multi-objective optimization algorithms, the dynamic weighted aggregation (DWA) method and the elitist non-dominated sorting genetic algorithm (NSGA-II) are used and compared. An evolutionary multi-objective approach to neural network regularization has a number of advantages compared to the traditional methods. First, a number of models with a spectrum of model complexity can be obtained in one optimization run instead of only one single solution. Second, an efficient new regularization term can be introduced, which is not applicable to gradient-based learning algorithms. As a natural by-product of the multi-objective optimization approach to neural network regularization, neural network ensembles can be easily constructed using the obtained networks with different levels of model complexity. Thus, the model complexity of the ensemble can be adjusted by adjusting the weight of each member network in the ensemble. Simulations are carried out on a test function to illustrate the feasibility of the proposed ideas.
---
paper_title: MOLeCS: Using Multiobjective Evolutionary Algorithms for Learning

paper_content:

MOLeCS is a classifier system (CS) which addresses its learning as a multiobjective task. Its aim is to develop an optimal set of rules, optimizing the accuracy and the generality of each rule simultaneously. This is achieved by considering these two goals in the rule fitness. The paper studies four multiobjective strategies that establish a compromise between accuracy and generality in different ways. The results suggest that including the decision maker's preferences in the search process improves the overall performance of the obtained rule set. The paper also studies a third major objective: covering (the maintenance of a set of different rules solving together the learning problem), through different niching mechanisms. After a performance analysis using some benchmark problems, MOLeCS is applied to a real-world categorization task: the diagnosis of breast cancer.
---
paper_title: Hierarchical Rank Density Genetic Algorithm for Radial-Basis Function Neural Network Design

paper_content:

In this paper, we propose a genetic algorithm based design procedure for a radial-basis function neural network. A Hierarchical Rank Density Genetic Algorithm (HRDGA) is used to evolve the neural network's topology and parameters simultaneously. Compared with traditional genetic algorithm based designs for neural networks, the hierarchical approach addresses several deficiencies highlighted in literature. In addition, the rank-density based fitness assignment technique is used to optimize the performance and topology of the evolved neural network to deal with the confliction between the training performance and network complexity. Instead of producing a single optimal solution, HRDGA provides a set of near-optimal neural networks to the designers so that they can have more flexibility for the final decision-making based on certain preferences. In terms of searching for a near-complete set of candidate networks with high performances, the networks designed by the proposed algorithm prove to be competitive,...
---
paper_title: Multi-objective structure selection for radial basis function networks based on genetic algorithm

paper_content:

Radial basis function (RBF) network is well known as a good performance approach to nonlinear system modeling. Though structure selection of RBF network is an important issue, the framework of this problem has not been established. In this paper, we propose multiobjective structure selection method for RBF networks based on MOGA (multiobjective genetic algorithm). The structure of RBF networks is encoded to the chromosomes in GA, then evolved toward to Pareto-optimum for multiobjective functions concerned with model accuracy and complexity. Some numerical simulation results indicate the applicability of the proposed approach.
---
paper_title: Automatic Generation of Controllers for Embodied Legged Organisms: A Pareto Evolutionary Multi-Objective Approach

paper_content:

In this paper, we investigate the use of a self-adaptive Pareto evolutionary multiobjective optimization (EMO) approach for evolving the controllers of virtual embodied organisms. The objective of this paper is to demonstrate the trade-off between quality of solutions and computational cost. We show empirically that evolving controllers using the proposed algorithm incurs significantly less computational cost when compared to a self-adaptive weighted sum EMO algorithm, a self-adaptive single-objective evolutionary algorithm (EA) and a hand-tuned Pareto EMO algorithm. The main contribution of the self-adaptive Pareto EMO approach is its ability to produce sufficiently good controllers with different locomotion capabilities in a single run, thereby reducing the evolutionary computational cost and allowing the designer to explore the space of good solutions simultaneously. Our results also show that self-adaptation was found to be highly beneficial in reducing redundancy when compared against the other algorithms. Moreover, it was also shown that genetic diversity was being maintained naturally by virtue of the system's inherent multi-objectivity.
---
paper_title: Generating fc fuzzy rule systems from data using evolution strategies

paper_content:

Sophisticated fuzzy rule systems are supposed to be flexible, complete, consistent and compact (FC/sup 3/). Flexibility, and consistency are essential for fuzzy systems to exhibit an excellent performance and to have a clear physical meaning, while compactness is crucial when the number of the input variables increases. However, the completeness and consistency conditions are often violated if a fuzzy system is generated from data collected from real world applications. A systematic design paradigm is proposed using evolution strategies. The structure of the fuzzy rules, which determines the compactness of the fuzzy systems, is evolved along with the parameters of the fuzzy systems. Special attention has been paid to the completeness and consistency of the rule base. The completeness is guaranteed by checking the completeness of the fuzzy partitioning of input variables and the completeness of the rule structure. An index of inconsistency is suggested with the help of a fuzzy similarity which can prevent the algorithm from generating rules that seriously contradict with each other or with the heuristic knowledge. In addition, soft T-norm and BADD defuzzification are introduced and optimized to increase the flexibility of the fuzzy system. The proposed approach is applied to the design of a distance controller for cars. It is verified that a FC/sup 3/ fuzzy system works very well both, for training and test driving situations, especially when the training data are insufficient.
---
paper_title: A Structure Identification Method of Submodels for Hierarchical Fuzzy Modeling Using the Multiple Objective Genetic Algorithm

paper_content:

Fuzzy models describe nonlinear input-output relationships with linguistic fuzzy rules. A hierarchical fuzzy modeling is promising for identification of fuzzy models of target systems that have many input variables. In the identification, (1) determination of a hierarchical structure of submodels, (2) selection of input variables of each submodel, (3) division of input and output space, (4) tuning of membership functions, and (5) determination of fuzzy inference method are carried out. This article presents a hierarchical fuzzy modeling method with an uneven division method of input space of each submodel. For selecting input variables of submodels, the multiple objective genetic algorithm (MOGA) is utilized. MOGA finds multiple models with different input variables and different numbers of fuzzy rules as compromising solutions. A human designer can choose desirable ones from these candidates. The proposed method is applied to acquisition of fuzzy rules from cyclists' pedaling data. In spite of a small number of data, the obtained model was able to give detailed suggestions to each cyclist. © 2002 Wiley Periodicals, Inc.
---
paper_title: Fuzzy Modeling of High-Dimensional Systems: Complexity Reduction and Interpretability Improvement

paper_content:

Fuzzy modeling of high-dimensional systems is a challenging topic. This paper proposes an effective approach to data-based fuzzy modeling of high-dimensional systems. An initial fuzzy rule system is generated based on the conclusion that optimal fuzzy rules cover extrema. Redundant rules are removed based on a fuzzy similarity measure. Then, the structure and parameters of the fuzzy system are optimized using a genetic algorithm and the gradient method. During optimization, rules that have a very low firing strength are deleted. Finally, interpretability of the fuzzy system is improved by fine training the fuzzy rules with regularization. The resulting fuzzy system generated by this method has the following distinct features: (1) the fuzzy system is quite simplified; (2) the fuzzy system is interpretable; and (3) the dependencies between the inputs and the output are clearly shown. This method has successfully been applied to a system that has 11 inputs and one output with 20 000 training data and 80 000 test data.
---
paper_title: Fuzzy modeling with multi-objective neuro-evolutionary algorithms

paper_content:

Interpretability aspects of fuzzy models have received quite some attention in recent years and may be obtained by using transparent rule-structures and well characterized fuzzy membership functions. Moreover, model compactness is important for the interpretability and is related to the number of rules and fuzzy sets. Besides these two criteria, the model accuracy should always be taken into account. In this way, several criteria appear in fuzzy modeling and then multi-objective evolutionary algorithms are suitable to capture several nondominated solutions in a single run of the algorithm. For fuzzy modeling, we describe a multi-objective neuro-evolutionary algorithm that considers all three objectives. The algorithm applies an accuracy criterium and a transparency criterium, based on fuzzy set similarity, while compactness is achieved by a specific technique, incorporated ad hoc within the evolutionary algorithm. Results are shown for an approximation problem studied before by other authors.
---
paper_title: Efficient fuzzy modeling under multiple criteria by using genetic algorithm

paper_content:

Fuzzy modeling is a method to describe input-output relationships of nonlinear systems. A genetic algorithm (GA) has been applied to fuzzy modeling for identification of the structure of a fuzzy model and selection of input variables. Trade-offs among multiple criteria make the search problem more complicated. For easy determination of weights on the criteria, a framework of model generation and testing was proposed by the authors. This framework divides the process of fuzzy modeling into two blocks, i.e. a model generation block and model testing block. The model generation block has criteria, with a higher degree of importance, and the model testing block has those with a lower degree of importance. In this paper, the idea of Pareto optimality is introduced to this framework and the effectiveness of the framework is examined by simulations.
---
paper_title: Multi-objective hierarchical genetic algorithm for interpretable fuzzy rule-based knowledge extraction

paper_content:

A new scheme based on multi-objective hierarchical genetic algorithm (MOHGA) is proposed to extract interpretable rule-based knowledge from data. The approach is derived from the use of multiple objective genetic algorithm (MOGA), where the genes of the chromosome are arranged into control genes and parameter genes. These genes are in a hierarchical form so that the control genes can manipulate the parameter genes in a more effective manner. The effectiveness of this chromosome formulation enables the fuzzy sets and rules to be optimally reduced. Some important concepts about the interpretability are introduced and the fitness function in the MOGA will consider both the accuracy and interpretability of the fuzzy model. In order to remove the redundancy of the rule base proactively, we further apply an interpretability-driven simplification method to newborn individuals. In our approach, we first apply the fuzzy clustering to generate an initial rule-based model. Then the multi-objective hierarchical genetic algorithm and the recursive least square method are used to obtain the optimized fuzzy models. The accuracy and the interpretability of fuzzy models derived by this approach are studied and presented in this paper. We compare our work with other methods reported in the literature on four examples: a synthetic nonlinear dynamic system, a nonlinear static system, the Lorenz system and the Mackey-Glass system. Simulation results show that the proposed approach is effective and practical in knowledge extraction.
---
paper_title: Advanced Fuzzy Systems Design and Applications

paper_content:

1. Fuzzy Sets and Fuzzy Systems.- 1.1 Basics of Fuzzy Sets.- 1.1.1 Fuzzy Sets.- 1.1.2 Fuzzy Operations.- 1.1.3 Fuzzy Relations.- 1.1.4 Measures of Fuzziness.- 1.1.5 Measures of Fuzzy Similarity.- 1.2 Fuzzy Rule Systems.- 1.2.1 Linguistic Variables and Linguistic Hedges.- 1.2.2 Fuzzy Rules for Modeling and Control.- 1.2.3 Mamdani Fuzzy Rule Systems.- 1.2.4 Takagi-Sugeno-Kang Fuzzy Rule Systems.- 1.2.5 Fuzzy Systems are Universal Approximators.- 1.3 Interpretability of Fuzzy Rule System.- 1.3.1 Introduction.- 1.3.2 The Properties of Membership Functions.- 1.3.3 Completeness of Fuzzy Partitions.- 1.3.4 Distinguishability of Fuzzy Partitions.- 1.3.5 Consistency of Fuzzy Rules.- 1.3.6 Completeness and Compactness of Rule Structure.- 1.4 Knowledge Processing with Fuzzy Logic.- 1.4.1 Knowledge Representation and Acquisition with IFTHEN Rules.- 1.4.2 Knowledge Representation with Fuzzy Preference Models.- 1.4.3 Fuzzy Group Decision Making.- 2. Evolutionary Algorithms.- 2.1 Introduction.- 2.2 Generic Evolutionary Algorithms.- 2.2.1 Representation.- 2.2.2 Recombination.- 2.2.3 Mutation.- 2.2.4 Selection.- 2.3 Adaptation and Self-Adaptation in Evolutionary Algorithms.- 2.3.1 Adaptation.- 2.3.2 Self-adaptation.- 2.4 Constraints Handling.- 2.5 Multi-objective Evolution.- 2.5.1 Weighted Aggregation Approaches.- 2.5.2 Population-based Non-Pareto Approaches.- 2.5.3 Pareto-based Approaches.- 2.5.4 Discussions.- 2.6 Evolution with Uncertain Fitness Functions.- 2.6.1 Noisy Fitness Functions.- 2.6.2 Approximate Fitness Functions.- 2.6.3 Robustness Considerations.- 2.7 Parallel Implementations.- 2.8 Summary.- 3. Artificial Neural Networks.- 3.1 Introduction.- 3.2 Feedforward Neural Network Models.- 3.2.1 Multilayer Perceptrons.- 3.2.2 Radial Basis Function Networks.- 3.3 Learning Algorithms.- 3.3.1 Supervised Learning.- 3.3.2 Unsupervised Learning.- 3.3.3 Reinforcement Learning.- 3.4 Improvement of Generalization.- 3.4.1 Heuristic Methods.- 3.4.2 Active Data Selection.- 3.4.3 Regularization.- 3.4.4 Network Ensembles.- 3.4.5 A Priori Knowledge.- 3.5 Rule Extraction from Neural Networks.- 3.5.1 Extraction of Symbolic Rules.- 3.5.2 Extraction of Fuzzy Rules.- 3.6 Interaction between Evolution and Learning.- 3.7 Summary.- 4. Conventional Data-driven Fuzzy Systems Design.- 4.1 Introduction.- 4.2 Fuzzy Inference Based Method.- 4.3 Wang-Mendel's Method.- 4.4 A Direct Method.- 4.5 An Adaptive Fuzzy Optimal Controller.- 4.6 Summary.- 5.Neural Network Based Fuzzy Systems Design.- 5.1 Neurofuzzy Systems.- 5.2 The Pi-sigma Neurofuzzy Model.- 5.2.1 The Takagi-Sugeno-Kang Fuzzy Model.- 5.2.2 The Hybrid Neural Network Model.- 5.2.3 Training Algorithms.- 5.2.4 Interpretability Issues.- 5.3 Modeling and Control Using the Neurofuzzy System.- 5.3.1 Short-term Precipitation Prediction.- 5.3.2 Dynamic Robot Control.- 5.4 Neurofuzzy Control of Nonlinear Systems.- 5.4.1 Fuzzy Linearization.- 5.4.2 Neurofuzzy Identification of the Subsystems.- 5.4.3 Design of Controller.- 5.4.4 Stability Analysis.- 5.5 Summary.- 6. Evolutionary Design of Fuzzy Systems.- 6.1 Introduction.- 6.2 Evolutionary Design of Flexible Structured Fuzzy Controller..- 6.2.1 A Flexible Structured Fuzzy Controller.- 6.2.2 Parameter Optimization Using Evolution Strategies...- 6.2.3 Simulation Study.- 6.3 Evolutionary Optimization of Fuzzy Rules.- 6.3.1 Genetic Coding of Fuzzy Systems.- 6.3.2 Fitness Function.- 6.3.3 Evolutionary Fuzzy Modeling of Robot Dynamics.- 6.4 Fuzzy Systems Design for High-Dimensional Systems.- 6.4.1 Curse of Dimensionality.- 6.4.2 Flexible Fuzzy Partitions.- 6.4.3 Hierarchical Structures.- 6.4.4 Input Dimension Reduction.- 6.4.5 GA-Based Input Selection.- 6.5 Summary.- 7. Knowledge Discovery by Extracting Interpretable Fuzzy Rules.- 7.1 Introduction.- 7.1.1 Data, Information and Knowledge.- 7.1.2 Interpretability and Knowledge Extraction.- 7.2 Evolutionary Interpretable Fuzzy Rule Generation.- 7.2.1 Evolution Strategy for Mixed Parameter Optimization.- 7.2.2 Genetic Representation of Fuzzy Systems.- 7.2.3 Multiobjective Fuzzy Systems Optimization.- 7.2.4 An Example: Fuzzy Vehicle Distance Controller.- 7.3 Interactive Co-evolution for Fuzzy Rule Extraction.- 7.3.1 Interactive Evolution.- 7.3.2 Co-evolution.- 7.3.3 Interactive Co-evolution of Interpretable Fuzzy Systems.- 7.4 Fuzzy Rule Extraction from RBF Networks.- 7.4.1 Radial-Basis-Function Networks and Fuzzy Systems.- 7.4.2 Fuzzy Rule Extraction by Regularization.- 7.4.3 Application Examples.- 7.5 Summary.- 8. Fuzzy Knowledge Incorporation into Neural Networks.- 8.1 Data and A Priori Knowledge.- 8.2 Knowledge Incorporation in Neural Networks for Control.- 8.2.1 Adaptive Inverse Neural Control.- 8.2.2 Knowledge Incorporation in Adaptive Neural Control.- 8.3 Fuzzy Knowledge Incorporation By Regularization.- 8.3.1 Knowledge Representation with Fuzzy Rules.- 8.3.2 Regularized Learning.- 8.4 Fuzzy Knowledge as A Related Task in Learning.- 8.4.1 Learning Related Tasks.- 8.4.2 Fuzzy Knowledge as A Related Task.- 8.5 Simulation Studies.- 8.5.1 Regularized Learning.- 8.5.2 Multi-task Learning.- 8.6 Summary.- 9. Fuzzy Preferences Incorporation into Multi-objective Optimization.- 9.1 Multi-objective Optimization and Preferences Handling.- 9.1.1 Multi-objective Optimization.- 9.1.2 Incorporation of Fuzzy Preferences.- 9.2 Evolutionary Dynamic Weighted Aggregation.- 9.2.1 Conventional Weighted Aggregation for MOO.- 9.2.2 Dynamically Weighted Aggregation.- 9.2.3 Archiving of Pareto Solutions.- 9.2.4 Simulation Studies.- 9.2.5 Theoretical Analysis.- 9.3 Fuzzy Preferences Incorporation in MOO.- 9.3.1 Converting Fuzzy Preferences into Crisp Weights.- 9.3.2 Converting Fuzzy Preferences into Weight Intervals.- 9.4 Summary.- References.
---
paper_title: Evolutionary multiobjective optimization for generating an ensemble of fuzzy rule-based classifiers

paper_content:

One advantage of evolutionary multiobjective optimization (EMO) algorithms over classical approaches is that many non-dominated solutions can be simultaneously obtained by their single run. In this paper, we propose an idea of using EMO algorithms for constructing an ensemble of fuzzy rule-based classifiers with high diversity. The classification of new patterns is performed based on the vote of multiple classifiers generated by a single run of EMO algorithms. Even when the classification performance of individual classifiers is not high, their ensemble often works well. The point is to generate multiple classifiers with high diversity. We demonstrate the ability of EMO algorithms to generate various non-dominated fuzzy rule-based classifiers with high diversity by their single run. Through computational experiments on some well-known benchmark data sets, it is shown that the vote of generated fuzzy rule-based classifiers leads to high classification performance on test patterns.
---
paper_title: Neural network ensembles

paper_content:

Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. >
---
paper_title: Neural network regularization and ensembling using multi-objective evolutionary algorithms

paper_content:

Regularization is an essential technique to improve generalization of neural networks. Traditionally, regularization is conducted by including an additional term in the cost function of a learning algorithm. One main drawback of these regularization techniques is that a hyperparameter that determines to which extension the regularization influences the learning algorithm must be determined beforehand. This paper addresses the neural network regularization problem from a multi-objective optimization point of view. During the optimization, both structure and parameters of the neural network will be optimized. A slightly modified version of two multi-objective optimization algorithms, the dynamic weighted aggregation (DWA) method and the elitist non-dominated sorting genetic algorithm (NSGA-II) are used and compared. An evolutionary multi-objective approach to neural network regularization has a number of advantages compared to the traditional methods. First, a number of models with a spectrum of model complexity can be obtained in one optimization run instead of only one single solution. Second, an efficient new regularization term can be introduced, which is not applicable to gradient-based learning algorithms. As a natural by-product of the multi-objective optimization approach to neural network regularization, neural network ensembles can be easily constructed using the obtained networks with different levels of model complexity. Thus, the model complexity of the ensemble can be adjusted by adjusting the weight of each member network in the ensemble. Simulations are carried out on a test function to illustrate the feasibility of the proposed ideas.
---
paper_title: Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy

paper_content:

Diversity among the members of a team of classifiers is deemed to be a key issue in classifier combination. However, measuring diversity is not straightforward because there is no generally accepted formal definition. We have found and studied ten statistics which can measure diversity among binary classifier outputs (correct or incorrect vote for the class label): four averaged pairwise measures (the Q statistic, the correlation, the disagreement and the double fault) and six non-pairwise measures (the entropy of the votes, the difficulty index, the Kohavi-Wolpert variance, the interrater agreement, the generalized diversity, and the coincident failure diversity). Four experiments have been designed to examine the relationship between the accuracy of the team and the measures of diversity, and among the measures themselves. Although there are proven connections between diversity and accuracy in some special cases, our results raise some doubts about the usefulness of diversity measures in building classifier ensembles in real-life pattern recognition problems.
---
paper_title: COVNET: a cooperative coevolutionary model for evolving artificial neural networks

paper_content:

This paper presents COVNET, a new cooperative coevolutionary model for evolving artificial neural networks. This model is based on the idea of coevolving subnetworks that must cooperate to form a solution for a specific problem, instead of evolving complete networks. The combination of this subnetworks is part of a coevolutionary process. The best combinations of subnetworks must be evolved together with the coevolution of the subnetworks. Several subpopulations of subnetworks coevolve cooperatively and genetically isolated. The individual of every subpopulation are combined to form whole networks. This is a different approach from most current models of evolutionary neural networks which try to develop whole networks. COVNET places as few restrictions as possible over the network structure, allowing the model to reach a wide variety of architectures during the evolution and to be easily extensible to other kind of neural networks. The performance of the model in solving three real problems of classification is compared with a modular network, the adaptive mixture of experts and with the results presented in the bibliography. COVNET has shown better generalization and produced smaller networks than the adaptive mixture of experts and has also achieved results, at least, comparable with the results in the bibliography.
---
paper_title: DIVACE: Diverse and Accurate Ensemble Learning Algorithm

paper_content:

In order for a neural network ensemble to generalise properly, two factors are considered vital. One is the diversity and the other is the accuracy of the networks that comprise the ensemble. There exists a tradeoff as to what should be the optimal measures of diversity and accuracy. The aim of this paper is to address this issue. We propose the DIVACE algorithm which tries to produce an ensemble as it searches for the optimum point on the diversity-accuracy curve. The DIVACE algorithm formulates the ensemble learning problem as a multi-objective problem explicitly.
---
paper_title: Diversity creation methods: a survey and categorisation

paper_content:

Ensemble approaches to classication and regression have attracted a great deal of interest in recent years. These methods can be shown both theoretically and empirically to outperform single predictors on a wide range of tasks. One of the elements required for accurate prediction when using an ensemble is recognised to be error \diversity". However, the exact meaning of this concept is not clear from the literature, particularly for classication tasks. In this paper we rst review the varied attempts to provide a formal explanation of error diversity, including several heuristic and qualitative explanations in the literature. For completeness of discussion we include not only the classication literature but also some excerpts of the rather more mature regression literature, which we believe can still provide some insights. We proceed to survey the various techniques used for creating diverse ensembles, and categorise them, forming a preliminary taxonomy of diversity creation methods. As part of this taxonomy we introduce the idea of implicit and explicit diversity creation methods, and three dimensions along which these may be applied. Finally we propose some new directions that may prove fruitful in understanding classication error diversity.
---
paper_title: Ensemble Learning via Negative Correlation

paper_content:

This paper presents a learning approach, i.e. negative correlation learning, for neural network ensembles. Unlike previous learning approaches for neural network ensembles, negative correlation learning attempts to train individual networks in an ensemble and combines them in the same learning process. In negative correlation learning, all the individual networks in the ensemble are trained simultaneously and interactively through the correlation penalty terms in their error functions. Rather than producing unbiased individual networks whose errors are uncorrelated, negative correlation learning can create negatively correlated networks to encourage specialisation and cooperation among the individual networks. Empirical studies have been carried out to show why and how negative correlation learning works. The experimental results show that negative correlation learning can produce neural network ensembles with good generalisation ability.
---
paper_title: Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization

paper_content:

In this paper, we present a comparison between two multiobjective formulations to the formation of neuro-ensembles. The first formulation splits the training set into two nonoverlapping stratified subsets and form an objective to minimize the training error on each subset, while the second formulation adds random noise to the training set to form a second objective. A variation of the memetic Pareto artificial neural network (MPANN) algorithm is used. MPANN is based on differential evolution for continuous optimization. The ensemble is formed from all networks on the Pareto frontier. It is found that the first formulation outperformed the second. The first formulation is also found to be competitive to other methods in the literature.
---
paper_title: Generating Accurate and Diverse Members of a Neural-Network Ensemble

paper_content:

Neural-network ensembles have been shown to be very accurate classification techniques. Previous work has shown that an effective ensemble should consist of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well. Most existing techniques, however, only indirectly address the problem of creating such a set of networks. In this paper we present a technique called ADDEMUP that uses genetic algorithms to directly search for an accurate and diverse set of trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to continually create new networks, keeping the set of networks that are as accurate as possible while disagreeing with each other as much as possible. Experiments on three DNA problems show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing approaches. Experiments also show that ADDEMUP is able to effectively incorporate prior knowledge, if available, to improve the quality of its ensemble.
---
paper_title: Ensemble Learning using Decorrelated Neural Networks

paper_content:

We describe a decorrelation network training method for improving the quality of regression learning in 'ensemble' neural networks NNs that are composed of linear combinations of individual NNs. In this method, individual networks are trained by backpropogation not only to reproduce a desired output, but also to have their errors linearly decorrelated with the other networks. Outputs from the individual networks are then linearly combined to produce the output of the ensemble network. We demonstrate the performances of decorrelated network training on learning the 'three-parity' logic function, a noisy sine function and a one-dimensional non-linear function, and compare the results with the ensemble networks composed of independently trained individual networks without decorrelation training . Empirical results show than when individual networks are forced to be decorrelated with one another the resulting ensemble NNs have lower mean squared errors than the ensemble networks having independently trained i...
---
paper_title: Design of a neural controller using multiobjective optimization for nonminimum phase systems

paper_content:

This paper presents a control architecture with a neural controller and a conventional linear controller for nonminimum phase systems. The objective is to minimize overall position errors as well as to maintain small undershooting. These attributes make it difficult to obtain the optimal solution which satisfied all individual objectives. Moreover, heuristic attempts of a proper combination of several objectives may produce a feasible solution but not necessarily an optimal one. With the concept of Pareto optimality and evolutionary programming, we train the controller more effectively and obtain a valuable set of optimal solutions. According to the preference, we can easily determine the most suitable solution from a pool of optimal candidates.
---
paper_title: Feature Selection for Ensembles: A Hierarchical Multi-Objective Genetic Algorithm Approach

paper_content:

Feature selection for ensembles has shown to be an effectivestrategy for ensemble creation. In this paper we presentan ensemble feature selection approach based on a hierarchicalmulti-objective genetic algorithm. The first level performsfeature selection in order to generate a set of goodclassifiers while the second one combines them to providea set of powerful ensembles. The proposed method is evaluatedin the context of handwritten digit recognition, usingthree different feature sets and neural networks (MLP) asclassifiers. Experiments conducted on NIST SD19 demonstratedthe effectiveness of the proposed strategy.
---
paper_title: Multi-class ROC analysis from a multi-objective optimisation perspective

paper_content:

The receiver operating characteristic (ROC) has become a standard tool for the analysis and comparison of classifiers when the costs of misclassification are unknown. There has been relatively little work, however, examining ROC for more than two classes. Here we discuss and present an extension to the standard two-class ROC for multi-class problems. We define the ROC surface for the Q-class problem in terms of a multi-objective optimisation problem in which the goal is to simultaneously minimise the Q(Q-1) misclassification rates, when the misclassification costs and parameters governing the classifier's behaviour are unknown. We present an evolutionary algorithm to locate the Pareto front-the optimal trade-off surface between misclassifications of different types. The use of the Pareto optimal surface to compare classifiers is discussed and we present a straightforward multi-class analogue of the Gini coefficient. The performance of the evolutionary algorithm is illustrated on a synthetic three class problem, for both k-nearest neighbour and multi-layer perceptron classifiers.
---
paper_title: Generalization Improvement in Multi-Objective Learning

paper_content:

Several heuristic methods have been suggested for improving the generalization capability in neural network learning, most of which are concerned with a single-objective (SO) learning tasks. In this work, we discuss generalization improvement in multi-objective learning (MO). As a case study, we investigate the generation of neural network classifiers based on the receiver operating characteristics (ROC) analysis using an evolutionary multi-objective optimization algorithm. We show on a few benchmark problems that for MO learning such as the ROC based classification, the generalization ability can be more efficiently improved within a multi-objective framework than within a single-objective one.
---
paper_title: Exploiting the trade-off — the benefits of multiple objectives in data clustering

paper_content:

In previous work, we have proposed a novel approach to data clustering based on the explicit optimization of a partitioning with respect to two complementary clustering objectives [6]. Here, we extend this idea by describing an advanced multiobjective clustering algorithm, MOCK, with the capacity to identify good solutions from the Pareto front, and to automatically determine the number of clusters in a data set. The algorithm has been subject to a thorough comparison with alternative clustering techniques and we briefly summarize these results. We then present investigations into the mechanisms at the heart of MOCK: we discuss a simple example demonstrating the synergistic effects at work in multiobjective clustering, which explain its superiority to single-objective clustering techniques, and we analyse how MOCK's Pareto fronts compare to the performance curves obtained by single-objective algorithms run with a range of different numbers of clusters specified.
---
paper_title: Evolutionary model selection in unsupervised learning

paper_content:

Feature subset selection is important not only for the insight gained from determining relevant modeling variables but also for the improved understandability, scalability, and possibly, accuracy of the resulting models. Feature selection has traditionally been studied in supervised learning situations, with some estimate of accuracy used to evaluate candidate subsets. However, we often cannot apply supervised learning for lack of a training signal. For these cases, we propose a new feature selection approach based on clustering. A number of heuristic criteria can be used to estimate the quality of clusters built from a given feature subset. Rather than combining such criteria, we use ELSA, an evolutionary local selection algorithm that maintains a diverse population of solutions that approximate the Pareto front in a multi-dimensional objective space. Each evolved solution represents a feature subset and a number of clusters; two representative clustering algorithms, K-means and EM, are applied to form the given number of clusters based on the selected features. Experimental results on both real and synthetic data show that the method can consistently find approximate Pareto-optimal solutions through which we can identify the significant features and an appropriate number of clusters. This results in models with better and clearer semantic relevance.
---
paper_title: Empirical evaluation of the improved Rprop learning algorithms

paper_content:

Abstract The Rprop algorithm proposed by Riedmiller and Braun is one of the best performing first-order learning methods for neural networks. We discuss modifications of this algorithm that improve its learning speed. The new optimization methods are empirically compared to the existing Rprop variants, the conjugate gradient method, Quickprop, and the BFGS algorithm on a set of neural network benchmark problems. The improved Rprop outperforms the other methods; only the BFGS performs better in the later stages of learning on some of the test problems. For the analysis of the local search behavior, we compare the Rprop algorithms on general hyperparabolic error landscapes, where the new variants confirm their improvement.
---
paper_title: A direct adaptive method for faster backpropagation learning: the RPROP algorithm

paper_content:

A learning algorithm for multilayer feedforward networks, RPROP (resilient propagation), is proposed. To overcome the inherent disadvantages of pure gradient-descent, RPROP performs a local adaptation of the weight-updates according to the behavior of the error function. Contrary to other adaptive techniques, the effect of the RPROP adaptation process is not blurred by the unforeseeable influence of the size of the derivative, but only dependent on the temporal behavior of its sign. This leads to an efficient and transparent adaptation process. The capabilities of RPROP are shown in comparison to other adaptive techniques. >
---
paper_title: Evolving artificial neural networks

paper_content:

Learning and evolution are two fundamental forms of adaptation. There has been a great interest in combining learning and evolution with artificial neural networks (ANNs) in recent years. This paper: 1) reviews different combinations between ANNs and evolutionary algorithms (EAs), including using EAs to evolve ANN connection weights, architectures, learning rules, and input features; 2) discusses different search operators which have been used in various EAs; and 3) points out possible future research directions. It is shown, through a considerably large literature review, that combinations between ANNs and EAs can lead to significantly better intelligent systems than relying on ANNs or EAs alone.
---
paper_title: A Fast Elitist Non-Dominated Sorting Genetic Algorithm for Multi-Objective Optimization : NSGA-II

paper_content:

Multi-objective evolutionary algorithms which use non-dominated sorting and sharing have been mainly criticized for their (i) O(MN3) computational complexity (where M is the number of objectives and N is the population size), (ii) non-elitism approach, and (iii) the need for specifying a sharing parameter. In this paper, we suggest a non-dominated sorting based multi-objective evolutionary algorithm (we called it the Non-dominated Sorting GA-II or NSGA-II) which alleviates all the above three difficulties. Specifically, a fast non-dominated sorting approach with O(MN2) computational complexity is presented. Second, a selection operator is presented which creates a mating pool by combining the parent and child populations and selecting the best (with respect to fitness and spread) N solutions. Simulation results on five difficult test problems show that the proposed NSGA-II, in most problems, is able to find much better spread of solutions and better convergence near the true Pareto-optimal front compared to PAES and SPEA--two other elitist multi-objective EAs which pay special attention towards creating a diverse Pareto-optimal front. Because of NSGA-II's low computational requirements, elitist approach, and parameter-less sharing approach, NSGA-II should find increasing applications in the years to come.
---
paper_title: Extraction of Symbolic Rules from Artificial Neural Networks

paper_content:

Although backpropagation ANNs generally predict better than decision trees do for pattern classification problems, they are often regarded as black boxes, i.e., their predictions cannot be explained as those of decision trees. In many applications, it is desirable to extract knowledge from trained ANNs for the users to gain a better understanding of how the networks solve the problems. A new rule extraction algorithm, called rule extraction from artificial neural networks (REANN) is proposed and implemented to extract symbolic rules from ANNs. A standard three-layer feedforward ANN is the basis of the algorithm. A four-phase training algorithm is proposed for backpropagation learning. Explicitness of the extracted rules is supported by comparing them to the symbolic rules generated by other methods. Extracted rules are comparable with other methods in terms of number of rules, average number of conditions for a rule, and predictive accuracy. Extensive experimental studies on several benchmarks classification problems, such as breast cancer, iris, diabetes, and season classification problems, demonstrate the effectiveness of the proposed approach with good generalization ability.
---
paper_title: Neural network regularization and ensembling using multi-objective evolutionary algorithms

paper_content:

Regularization is an essential technique to improve generalization of neural networks. Traditionally, regularization is conducted by including an additional term in the cost function of a learning algorithm. One main drawback of these regularization techniques is that a hyperparameter that determines to which extension the regularization influences the learning algorithm must be determined beforehand. This paper addresses the neural network regularization problem from a multi-objective optimization point of view. During the optimization, both structure and parameters of the neural network will be optimized. A slightly modified version of two multi-objective optimization algorithms, the dynamic weighted aggregation (DWA) method and the elitist non-dominated sorting genetic algorithm (NSGA-II) are used and compared. An evolutionary multi-objective approach to neural network regularization has a number of advantages compared to the traditional methods. First, a number of models with a spectrum of model complexity can be obtained in one optimization run instead of only one single solution. Second, an efficient new regularization term can be introduced, which is not applicable to gradient-based learning algorithms. As a natural by-product of the multi-objective optimization approach to neural network regularization, neural network ensembles can be easily constructed using the obtained networks with different levels of model complexity. Thus, the model complexity of the ensemble can be adjusted by adjusting the weight of each member network in the ensemble. Simulations are carried out on a test function to illustrate the feasibility of the proposed ideas.
---
paper_title: DIVACE: Diverse and Accurate Ensemble Learning Algorithm

paper_content:

In order for a neural network ensemble to generalise properly, two factors are considered vital. One is the diversity and the other is the accuracy of the networks that comprise the ensemble. There exists a tradeoff as to what should be the optimal measures of diversity and accuracy. The aim of this paper is to address this issue. We propose the DIVACE algorithm which tries to produce an ensemble as it searches for the optimum point on the diversity-accuracy curve. The DIVACE algorithm formulates the ensemble learning problem as a multi-objective problem explicitly.
---
paper_title: Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization

paper_content:

In this paper, we present a comparison between two multiobjective formulations to the formation of neuro-ensembles. The first formulation splits the training set into two nonoverlapping stratified subsets and form an objective to minimize the training error on each subset, while the second formulation adds random noise to the training set to form a second objective. A variation of the memetic Pareto artificial neural network (MPANN) algorithm is used. MPANN is based on differential evolution for continuous optimization. The ensemble is formed from all networks on the Pareto frontier. It is found that the first formulation outperformed the second. The first formulation is also found to be competitive to other methods in the literature.
---
paper_title: Extracting Rules from Artificial Neural Networks with Distributed Representations

paper_content:

Although artificial neural networks have been applied in a variety of real-world scenarios with remarkable success, they have often been criticized for exhibiting a low degree of human comprehensibility. Techniques that compile compact sets of symbolic rules out of artificial neural networks offer a promising perspective to overcome this obvious deficiency of neural network representations. ::: ::: This paper presents an approach to the extraction of if-then rules from artificial neural networks. Its key mechanism is validity interval analysis, which is a generic tool for extracting symbolic knowledge by propagating rule-like knowledge through Backpropagation-style neural networks. Empirical studies in a robot arm domain illustrate the appropriateness of the proposed method for extracting rules from networks with real-valued and distributed representations.
---
paper_title: Fuzzy Modeling of High-Dimensional Systems: Complexity Reduction and Interpretability Improvement

paper_content:

Fuzzy modeling of high-dimensional systems is a challenging topic. This paper proposes an effective approach to data-based fuzzy modeling of high-dimensional systems. An initial fuzzy rule system is generated based on the conclusion that optimal fuzzy rules cover extrema. Redundant rules are removed based on a fuzzy similarity measure. Then, the structure and parameters of the fuzzy system are optimized using a genetic algorithm and the gradient method. During optimization, rules that have a very low firing strength are deleted. Finally, interpretability of the fuzzy system is improved by fine training the fuzzy rules with regularization. The resulting fuzzy system generated by this method has the following distinct features: (1) the fuzzy system is quite simplified; (2) the fuzzy system is interpretable; and (3) the dependencies between the inputs and the output are clearly shown. This method has successfully been applied to a system that has 11 inputs and one output with 20 000 training data and 80 000 test data.
---
paper_title: Evolutionary Multi-objective Optimization for Simultaneous Generation of Signal-type and Symbol-type Representations

paper_content:

It has been a controversial issue in the research of cognitive science and artificial intelligence whether signal-type representations (typically connectionist networks) or symbol-type representations (e.g., semantic networks, production systems) should be used. Meanwhile, it has also been recognized that both types of information representations might exist in the human brain. In addition, symbol-type representations are often very helpful in gaining insights into unknown systems. For these reasons, comprehensible symbolic rules need to be extracted from trained neural networks. In this paper, an evolutionary multi-objective algorithm is employed to generate multiple models that facilitate the generation of signal-type and symbol-type representations simultaneously. It is argued that one main difference between signal-type and symbol-type representations lies in the fact that the signal-type representations are models of a higher complexity (fine representation), whereas symbol-type representations are models of a lower complexity (coarse representation). Thus, by generating models with a spectrum of model complexity, we are able to obtain a population of models of both signal-type and symbol-type quality, although certain post-processing is needed to get a fully symbol-type representation. An illustrative example is given on generating neural networks for the breast cancer diagnosis benchmark problem.
---
paper_title: Advanced Fuzzy Systems Design and Applications

paper_content:

1. Fuzzy Sets and Fuzzy Systems.- 1.1 Basics of Fuzzy Sets.- 1.1.1 Fuzzy Sets.- 1.1.2 Fuzzy Operations.- 1.1.3 Fuzzy Relations.- 1.1.4 Measures of Fuzziness.- 1.1.5 Measures of Fuzzy Similarity.- 1.2 Fuzzy Rule Systems.- 1.2.1 Linguistic Variables and Linguistic Hedges.- 1.2.2 Fuzzy Rules for Modeling and Control.- 1.2.3 Mamdani Fuzzy Rule Systems.- 1.2.4 Takagi-Sugeno-Kang Fuzzy Rule Systems.- 1.2.5 Fuzzy Systems are Universal Approximators.- 1.3 Interpretability of Fuzzy Rule System.- 1.3.1 Introduction.- 1.3.2 The Properties of Membership Functions.- 1.3.3 Completeness of Fuzzy Partitions.- 1.3.4 Distinguishability of Fuzzy Partitions.- 1.3.5 Consistency of Fuzzy Rules.- 1.3.6 Completeness and Compactness of Rule Structure.- 1.4 Knowledge Processing with Fuzzy Logic.- 1.4.1 Knowledge Representation and Acquisition with IFTHEN Rules.- 1.4.2 Knowledge Representation with Fuzzy Preference Models.- 1.4.3 Fuzzy Group Decision Making.- 2. Evolutionary Algorithms.- 2.1 Introduction.- 2.2 Generic Evolutionary Algorithms.- 2.2.1 Representation.- 2.2.2 Recombination.- 2.2.3 Mutation.- 2.2.4 Selection.- 2.3 Adaptation and Self-Adaptation in Evolutionary Algorithms.- 2.3.1 Adaptation.- 2.3.2 Self-adaptation.- 2.4 Constraints Handling.- 2.5 Multi-objective Evolution.- 2.5.1 Weighted Aggregation Approaches.- 2.5.2 Population-based Non-Pareto Approaches.- 2.5.3 Pareto-based Approaches.- 2.5.4 Discussions.- 2.6 Evolution with Uncertain Fitness Functions.- 2.6.1 Noisy Fitness Functions.- 2.6.2 Approximate Fitness Functions.- 2.6.3 Robustness Considerations.- 2.7 Parallel Implementations.- 2.8 Summary.- 3. Artificial Neural Networks.- 3.1 Introduction.- 3.2 Feedforward Neural Network Models.- 3.2.1 Multilayer Perceptrons.- 3.2.2 Radial Basis Function Networks.- 3.3 Learning Algorithms.- 3.3.1 Supervised Learning.- 3.3.2 Unsupervised Learning.- 3.3.3 Reinforcement Learning.- 3.4 Improvement of Generalization.- 3.4.1 Heuristic Methods.- 3.4.2 Active Data Selection.- 3.4.3 Regularization.- 3.4.4 Network Ensembles.- 3.4.5 A Priori Knowledge.- 3.5 Rule Extraction from Neural Networks.- 3.5.1 Extraction of Symbolic Rules.- 3.5.2 Extraction of Fuzzy Rules.- 3.6 Interaction between Evolution and Learning.- 3.7 Summary.- 4. Conventional Data-driven Fuzzy Systems Design.- 4.1 Introduction.- 4.2 Fuzzy Inference Based Method.- 4.3 Wang-Mendel's Method.- 4.4 A Direct Method.- 4.5 An Adaptive Fuzzy Optimal Controller.- 4.6 Summary.- 5.Neural Network Based Fuzzy Systems Design.- 5.1 Neurofuzzy Systems.- 5.2 The Pi-sigma Neurofuzzy Model.- 5.2.1 The Takagi-Sugeno-Kang Fuzzy Model.- 5.2.2 The Hybrid Neural Network Model.- 5.2.3 Training Algorithms.- 5.2.4 Interpretability Issues.- 5.3 Modeling and Control Using the Neurofuzzy System.- 5.3.1 Short-term Precipitation Prediction.- 5.3.2 Dynamic Robot Control.- 5.4 Neurofuzzy Control of Nonlinear Systems.- 5.4.1 Fuzzy Linearization.- 5.4.2 Neurofuzzy Identification of the Subsystems.- 5.4.3 Design of Controller.- 5.4.4 Stability Analysis.- 5.5 Summary.- 6. Evolutionary Design of Fuzzy Systems.- 6.1 Introduction.- 6.2 Evolutionary Design of Flexible Structured Fuzzy Controller..- 6.2.1 A Flexible Structured Fuzzy Controller.- 6.2.2 Parameter Optimization Using Evolution Strategies...- 6.2.3 Simulation Study.- 6.3 Evolutionary Optimization of Fuzzy Rules.- 6.3.1 Genetic Coding of Fuzzy Systems.- 6.3.2 Fitness Function.- 6.3.3 Evolutionary Fuzzy Modeling of Robot Dynamics.- 6.4 Fuzzy Systems Design for High-Dimensional Systems.- 6.4.1 Curse of Dimensionality.- 6.4.2 Flexible Fuzzy Partitions.- 6.4.3 Hierarchical Structures.- 6.4.4 Input Dimension Reduction.- 6.4.5 GA-Based Input Selection.- 6.5 Summary.- 7. Knowledge Discovery by Extracting Interpretable Fuzzy Rules.- 7.1 Introduction.- 7.1.1 Data, Information and Knowledge.- 7.1.2 Interpretability and Knowledge Extraction.- 7.2 Evolutionary Interpretable Fuzzy Rule Generation.- 7.2.1 Evolution Strategy for Mixed Parameter Optimization.- 7.2.2 Genetic Representation of Fuzzy Systems.- 7.2.3 Multiobjective Fuzzy Systems Optimization.- 7.2.4 An Example: Fuzzy Vehicle Distance Controller.- 7.3 Interactive Co-evolution for Fuzzy Rule Extraction.- 7.3.1 Interactive Evolution.- 7.3.2 Co-evolution.- 7.3.3 Interactive Co-evolution of Interpretable Fuzzy Systems.- 7.4 Fuzzy Rule Extraction from RBF Networks.- 7.4.1 Radial-Basis-Function Networks and Fuzzy Systems.- 7.4.2 Fuzzy Rule Extraction by Regularization.- 7.4.3 Application Examples.- 7.5 Summary.- 8. Fuzzy Knowledge Incorporation into Neural Networks.- 8.1 Data and A Priori Knowledge.- 8.2 Knowledge Incorporation in Neural Networks for Control.- 8.2.1 Adaptive Inverse Neural Control.- 8.2.2 Knowledge Incorporation in Adaptive Neural Control.- 8.3 Fuzzy Knowledge Incorporation By Regularization.- 8.3.1 Knowledge Representation with Fuzzy Rules.- 8.3.2 Regularized Learning.- 8.4 Fuzzy Knowledge as A Related Task in Learning.- 8.4.1 Learning Related Tasks.- 8.4.2 Fuzzy Knowledge as A Related Task.- 8.5 Simulation Studies.- 8.5.1 Regularized Learning.- 8.5.2 Multi-task Learning.- 8.6 Summary.- 9. Fuzzy Preferences Incorporation into Multi-objective Optimization.- 9.1 Multi-objective Optimization and Preferences Handling.- 9.1.1 Multi-objective Optimization.- 9.1.2 Incorporation of Fuzzy Preferences.- 9.2 Evolutionary Dynamic Weighted Aggregation.- 9.2.1 Conventional Weighted Aggregation for MOO.- 9.2.2 Dynamically Weighted Aggregation.- 9.2.3 Archiving of Pareto Solutions.- 9.2.4 Simulation Studies.- 9.2.5 Theoretical Analysis.- 9.3 Fuzzy Preferences Incorporation in MOO.- 9.3.1 Converting Fuzzy Preferences into Crisp Weights.- 9.3.2 Converting Fuzzy Preferences into Weight Intervals.- 9.4 Summary.- References.
---
paper_title: Support-vector networks

paper_content:

Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
---
paper_title: Exploiting the trade-off — the benefits of multiple objectives in data clustering

paper_content:

In previous work, we have proposed a novel approach to data clustering based on the explicit optimization of a partitioning with respect to two complementary clustering objectives [6]. Here, we extend this idea by describing an advanced multiobjective clustering algorithm, MOCK, with the capacity to identify good solutions from the Pareto front, and to automatically determine the number of clusters in a data set. The algorithm has been subject to a thorough comparison with alternative clustering techniques and we briefly summarize these results. We then present investigations into the mechanisms at the heart of MOCK: we discuss a simple example demonstrating the synergistic effects at work in multiobjective clustering, which explain its superiority to single-objective clustering techniques, and we analyse how MOCK's Pareto fronts compare to the performance curves obtained by single-objective algorithms run with a range of different numbers of clusters specified.
---
paper_title: Pareto evolutionary neural networks

paper_content:

For the purposes of forecasting (or classification) tasks neural networks (NNs) are typically trained with respect to Euclidean distance minimization. This is commonly the case irrespective of any other end user preferences. In a number of situations, most notably time series forecasting, users may have other objectives in addition to Euclidean distance minimization. Recent studies in the NN domain have confronted this problem by propagating a linear sum of errors. However this approach implicitly assumes a priori knowledge of the error surface defined by the problem, which, typically, is not the case. This study constructs a novel methodology for implementing multiobjective optimization within the evolutionary neural network (ENN) domain. This methodology enables the parallel evolution of a population of ENN models which exhibit estimated Pareto optimality with respect to multiple error measures. A new method is derived from this framework, the Pareto evolutionary neural network (Pareto-ENN). The Pareto-ENN evolves a population of models that may be heterogeneous in their topologies inputs and degree of connectivity, and maintains a set of the Pareto optimal ENNs that it discovers. New generalization methods to deal with the unique properties of multiobjective error minimization that are not apparent in the uni-objective case are presented and compared on synthetic data, with a novel method based on bootstrapping of the training data shown to significantly improve generalization ability. Finally experimental evidence is presented in this study demonstrating the general application potential of the framework by generating populations of ENNs for forecasting 37 different international stock indexes.
---
paper_title: Sparse coding with an overcomplete basis set: A strategy employed by V1?

paper_content:

The spatial receptive fields of simple cells in mammalian striate cortex have been reasonably well described physiologically and can be characterized as being localized, oriented, and ban@ass, comparable with the basis functions of wavelet transforms. Previously, we have shown that these receptive field properties may be accounted for in terms of a strategy for producing a sparse distribution of output activity in response to natural images. Here, in addition to describing this work in a more expansive fashion, we examine the neurobiological implications of sparse coding. Of particular interest is the case when the code is overcomplete--i.e., when the number of code elements is greater than the effective dimensionality of the input space. Because the basis functions are non-orthogonal and not linearly independent of each other, sparsifying the code will recruit only those basis functions necessary for representing a given input, and so the input-output function will deviate from being purely linear. These deviations from linearity provide a potential explanation for the weak forms of non-linearity observed in the response properties of cortical simple cells, and they further make predictions about the expected interactions among units in response to naturalistic stimuli. © 1997 Elsevier Science Ltd
---
paper_title: Ensemble Learning via Negative Correlation

paper_content:

This paper presents a learning approach, i.e. negative correlation learning, for neural network ensembles. Unlike previous learning approaches for neural network ensembles, negative correlation learning attempts to train individual networks in an ensemble and combines them in the same learning process. In negative correlation learning, all the individual networks in the ensemble are trained simultaneously and interactively through the correlation penalty terms in their error functions. Rather than producing unbiased individual networks whose errors are uncorrelated, negative correlation learning can create negatively correlated networks to encourage specialisation and cooperation among the individual networks. Empirical studies have been carried out to show why and how negative correlation learning works. The experimental results show that negative correlation learning can produce neural network ensembles with good generalisation ability.
---
paper_title: Hierarchical Rank Density Genetic Algorithm for Radial-Basis Function Neural Network Design

paper_content:

In this paper, we propose a genetic algorithm based design procedure for a radial-basis function neural network. A Hierarchical Rank Density Genetic Algorithm (HRDGA) is used to evolve the neural network's topology and parameters simultaneously. Compared with traditional genetic algorithm based designs for neural networks, the hierarchical approach addresses several deficiencies highlighted in literature. In addition, the rank-density based fitness assignment technique is used to optimize the performance and topology of the evolved neural network to deal with the confliction between the training performance and network complexity. Instead of producing a single optimal solution, HRDGA provides a set of near-optimal neural networks to the designers so that they can have more flexibility for the final decision-making based on certain preferences. In terms of searching for a near-complete set of candidate networks with high performances, the networks designed by the proposed algorithm prove to be competitive,...
---
paper_title: Neural network regularization and ensembling using multi-objective evolutionary algorithms

paper_content:

Regularization is an essential technique to improve generalization of neural networks. Traditionally, regularization is conducted by including an additional term in the cost function of a learning algorithm. One main drawback of these regularization techniques is that a hyperparameter that determines to which extension the regularization influences the learning algorithm must be determined beforehand. This paper addresses the neural network regularization problem from a multi-objective optimization point of view. During the optimization, both structure and parameters of the neural network will be optimized. A slightly modified version of two multi-objective optimization algorithms, the dynamic weighted aggregation (DWA) method and the elitist non-dominated sorting genetic algorithm (NSGA-II) are used and compared. An evolutionary multi-objective approach to neural network regularization has a number of advantages compared to the traditional methods. First, a number of models with a spectrum of model complexity can be obtained in one optimization run instead of only one single solution. Second, an efficient new regularization term can be introduced, which is not applicable to gradient-based learning algorithms. As a natural by-product of the multi-objective optimization approach to neural network regularization, neural network ensembles can be easily constructed using the obtained networks with different levels of model complexity. Thus, the model complexity of the ensemble can be adjusted by adjusting the weight of each member network in the ensemble. Simulations are carried out on a test function to illustrate the feasibility of the proposed ideas.
---
paper_title: DIVACE: Diverse and Accurate Ensemble Learning Algorithm

paper_content:

In order for a neural network ensemble to generalise properly, two factors are considered vital. One is the diversity and the other is the accuracy of the networks that comprise the ensemble. There exists a tradeoff as to what should be the optimal measures of diversity and accuracy. The aim of this paper is to address this issue. We propose the DIVACE algorithm which tries to produce an ensemble as it searches for the optimum point on the diversity-accuracy curve. The DIVACE algorithm formulates the ensemble learning problem as a multi-objective problem explicitly.
---
paper_title: Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization

paper_content:

In this paper, we present a comparison between two multiobjective formulations to the formation of neuro-ensembles. The first formulation splits the training set into two nonoverlapping stratified subsets and form an objective to minimize the training error on each subset, while the second formulation adds random noise to the training set to form a second objective. A variation of the memetic Pareto artificial neural network (MPANN) algorithm is used. MPANN is based on differential evolution for continuous optimization. The ensemble is formed from all networks on the Pareto frontier. It is found that the first formulation outperformed the second. The first formulation is also found to be competitive to other methods in the literature.
---
paper_title: Generating Accurate and Diverse Members of a Neural-Network Ensemble

paper_content:

Neural-network ensembles have been shown to be very accurate classification techniques. Previous work has shown that an effective ensemble should consist of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well. Most existing techniques, however, only indirectly address the problem of creating such a set of networks. In this paper we present a technique called ADDEMUP that uses genetic algorithms to directly search for an accurate and diverse set of trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to continually create new networks, keeping the set of networks that are as accurate as possible while disagreeing with each other as much as possible. Experiments on three DNA problems show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing approaches. Experiments also show that ADDEMUP is able to effectively incorporate prior knowledge, if available, to improve the quality of its ensemble.
---
paper_title: Parameter convergence and learning curves for neural networks

paper_content:

We revisit the oft-studied asymptotic (in sample size) behavior of the parameter or weight estimate returned by any member of a large family of neural network training algorithms. By properly accounting for the characteristic property of neural networks that their empirical and generalization errors possess multiple minima, we establish conditions under which the parameter estimate converges strongly into the set of minima of the generalization error. These results are then used to derive learning curves for generalization and empirical errors that leads to bounds on rates of convergence.
---
paper_title: Parameter Incremental Learning Algorithm for Neural Networks

paper_content:

In this paper, a novel stochastic (or online) training algorithm for neural networks, named parameter incremental learning (PIL) algorithm, is proposed and developed. The main idea of the PIL strategy is that the learning algorithm should not only adapt to the newly presented input-output training pattern by adjusting parameters, but also preserve the prior results. A general PIL algorithm for feedforward neural networks is accordingly presented as the first-order approximate solution to an optimization problem, where the performance index is the combination of proper measures of preservation and adaptation. The PIL algorithms for the multilayer perceptron (MLP) are subsequently derived. Numerical studies show that for all the three benchmark problems used in this paper the PIL algorithm for MLP is measurably superior to the standard online backpropagation (BP) algorithm and the stochastic diagonal Levenberg-Marquardt (SDLM) algorithm in terms of the convergence speed and accuracy. Other appealing features of the PIL algorithm are that it is computationally as simple as the BP algorithm, and as easy to use as the BP algorithm. It, therefore, can be applied, with better performance, to any situations where the standard online BP algorithm is applicable
---
paper_title: Catastrophic forgetting in simple networks: an analysis of the pseudorehearsal solution

paper_content:

Catastrophic forgetting is a major problem for sequential learning in neural networks. One very general solution to this problem, known as 'pseudorehearsal', works well in practice for nonlinear networks but has not been analysed before. This paper formalizes pseudorehearsal in linear networks. We show that the method can fail in low dimensions but is guaranteed to succeed in high dimensions under fairly general conditions. In this case an optimal version of the method is equivalent to a simple modification of the 'delta rule'.
---
paper_title: Reducing local optima in single-objective problems by multi-objectivization, in

paper_content:

One common characterization of how simple hill-climbing optimization methods can fail is that they become trapped in local optima - a state where no small modification of the current best solution will produce a solution that is better. This measure of 'better' depends on the performance of the solution with respect to the single objective being optimized. In contrast, multi-objective optimization (MOO) involves the simultaneous optimization of a number of objectives. Accordingly, the multi-objective notion of 'better' permits consideration of solutions that may be superior in one objective but not in another. Intuitively, we may say that this gives a hill-climber in multi-objective space more freedom to explore and less likelihood of becoming trapped. In this paper, we investigate this intuition by comparing the performance of simple hill-climber-style algorithms on single-objective problems and multi-objective versions of those same problems. Using an abstract building-block problem we illustrate how 'multi-objectivizing' a single-objective optimization (SOO) problem can remove local optima. Then we investigate small instances of the travelling salesman problem where additional objectives are defined using arbitrary sub-tours. Results indicate that multi-objectivization can reduce local optima and facilitate improved optimization in some cases. These results enlighten our intuitions about the nature of search in multi-objective optimization and sources of difficulty in single-objective optimization.
---
paper_title: Classes of network connectivity and dynamics

paper_content:

Many kinds of complex systems exhibit characteristic patterns of temporal correlations that emerge as the result of functional interactions within a structured network. One such complex system is the brain, composed of numerous neuronal units linked by synaptic connections. The activity of these neuronal units gives rise to dynamic states that are characterized by specific patterns of neuronal activation and co-activation. These patterns, called functional connectivity, are possible neural correlates of perceptual and cognitive processes. Which functional connectivity patterns arise depends on the anatomical structure of the underlying network, which in turn is modified by a broad range of activity-dependent processes. Given this intricate relationship between structure and function, the question of how patterns of anatomical connectivity constrain or determine dynamical patterns is of considerable theoretical importance. The present study develops computational tools to analyze networks in terms of their structure and dynamics. We identify different classes of network, including networks that are characterized by high complexity. These highly complex networks have distinct structural characteristics such as clustered connectivity and short wiring length similar to those of large-scale networks of the cerebral cortex. 2002 Wiley Periodicals, Inc.
---
paper_title: CONNECTIVITY IN REAL AND EVOLVED ASSOCIATIVE MEMORIES

paper_content:

Finding efficient patterns of connectivity in sparse associative memories is a difficult problem. It is, however, one that real neuronal networks, such as the mammalian cortex, must have solved. We have investigated evolved computational models of sparsely connected associative memories and found that some patterns of connectivity produce both good performance and efficient use of resources. This could illuminate how real biological systems solve the problem.
---
paper_title: A universal theorem on learning curves

paper_content:

Abstract A learning curve shows how fast a learning machine improves its behavior as the number of training examples increases. This paper proves a universal asymptotic behavior of learning curves for general noiseless dichotomy machines, or neural networks. It is proved that irrespective of the architecture of a machine, the average predictive entropy or the information gain 〈 e ∗ ( t )〉 converges to 0 as 〈 e ∗ ( t )〉 ∼ d/t as the number t of training exampies increases, where d is the number of modifiable parameters of a machine.
---
paper_title: Evolutionary multi-objective optimization of spiking neural networks

paper_content:

Evolutionary multi-objective optimization of spiking neural networks for solving classification problems is studied in this paper. By means of a Paretobased multi-objective genetic algorithm, we are able to optimize both classification performance and connectivity of spiking neural networks with the latency coding. During optimization, the connectivity between two neurons, i.e., whether two neurons are connected, and if connected, both weight and delay between the two neurons, are evolved. We minimize the the classification error in percentage or the root mean square error for optimizing performance, and minimize the number of connections or the sum of delays for connectivity to investigate the influence of the objectives on the performance and connectivity of spiking neural networks. Simulation results on two benchmarks show that Pareto-based evolutionary optimization of spiking neural networks is able to offer a deeper insight into the properties of the spiking neural networks and the problem at hand.
---
paper_title: On-line learning with minimal degradation in feedforward networks

paper_content:

Dealing with nonstationary processes requires quick adaptation while at the same time avoiding catastrophic forgetting. A neural learning technique that satisfies these requirements, without sacrificing the benefits of distributed representations, is presented. It relies on a formalization of the problem as the minimization of the error over the previously learned input-output patterns, subject to the constraint of perfect encoding of the new pattern. Then this constrained optimization problem is transformed into an unconstrained one with hidden-unit activations as variables. This new formulation leads to an algorithm for solving the problem, which we call learning with minimal degradation (LMD). Some experimental comparisons of the performance of LMD with backpropagation are provided which, besides showing the advantages of using LMD, reveal the dependence of forgetting on the learning rate in backpropagation. We also explain why overtraining affects forgetting and fault tolerance, which are seen as related problems. >
---

Title: Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies

Section 1: Introduction
Description 1: Provide an introduction to machine learning and its objectives, the relevance of optimization in learning algorithms, and an overview of the paper structure.

Section 2: Single-Objective Learning
Description 2: Discuss learning algorithms which optimize only one objective function, with examples such as mean squared error in supervised learning and k-means clustering in unsupervised learning.

Section 3: Scalarized Multiobjective Learning
Description 3: Explain the concept of scalarized multiobjective learning where multiple objectives are combined into a single scalar objective function, and discuss its application and limitations.

Section 4: Pareto-Based Multiobjective Learning
Description 4: Introduce Pareto-based multiobjective learning, its principles of optimization, and how it does not require the prior specification of a hyperparameter like scalarized approaches.

Section 5: Earlier Ideas
Description 5: Review the initial attempts to apply the Pareto-based approach to supervision learning, highlighting key studies and developments.

Section 6: Generalization Improvement
Description 6: Discuss how Pareto-based learning can improve model generalization by balancing training error and network complexity or regularization terms.

Section 7: Interpretability Enhancement in Rule Extraction
Description 7: Examine the application of Pareto-based approaches for generating interpretable rules from data or trained models, focusing on fuzzy rule systems and logical rules.

Section 8: Diverse Ensemble Generation
Description 8: Explore the benefits of using Pareto-based approaches to generate diverse and accurate ensemble models, detailing methods to ensure diversity and effectiveness of the ensemble.

Section 9: Miscellaneous
Description 9: Highlight specific applications of Pareto-based multiobjective learning, such as supervised feature selection, systems control, and ROC curve generation.

Section 10: Multiobjective Unsupervised Learning
Description 10: Focus on the use of Pareto-based methods in unsupervised learning scenarios, primarily multiobjective data clustering, and their advantages.

Section 11: Case Studies: Results
Description 11: Present the results of the case studies performed using Pareto-based approaches, including identifying interpretable models, model selection, and creating neural network ensembles.

Section 12: Summary and Outlook
Description 12: Summarize the discussions, emphasize the advantages and potential of Pareto-based approaches in machine learning, and suggest directions for future research.