Diversity creation methods: a survey and categorisation
---
paper_title: Generalization error of ensemble estimators

paper_content:

It has been empirically shown that a better estimate with less generalization error can be obtained by averaging outputs of multiple estimators. This paper presents an analytical result for the generalization error of ensemble estimators. First, we derive a general expression of the ensemble generalization error by using factors of interest (bias, variance, covariance, and noise variance) and show how the generalization error is affected by each of them. Some special cases are then investigated. The result of a simulation is shown to verify our analytical result. A practically important problem of the ensemble approach, ensemble dilemma, is also discussed.
---
paper_title: Bagging Predictors

paper_content:

Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.
---
paper_title: Improving Regression Estimation: Averaging Methods for Variance Reduction with Extensions to General Convex Measure Optimization

paper_content:

A general theoretical framework for Monte Carlo averaging methods of improving regression estimates is presented with application to neural network classification and time series prediction. Given a population of regression estimators, it is shown how to construct a hybrid estimator which is as good as or better than, in the MSE sense, any estimator in the population. ::: It is argued that the ensemble method presented has several properties: It efficiently uses all the regressors of a population--none need be discarded. It efficiently uses all the available data for training without over-fitting. It inherently performs regularization by smoothing in functional space which helps to avoid over-fitting. It utilizes local minima to construct improved estimates whereas other regression algorithms are hindered by local minima. It is ideally suited for parallel computation. It leads to a very useful and natural measure of the number of distinct estimators in a population. The optimal parameters of the ensemble estimator are given in closed form. ::: It is shown that this result derives from the notion of convexity and can be applied to a wide variety of optimization algorithms including: Mean Square Error, a general class of $L\sb{p}$-norm cost functions, Maximum Likelihood Estimation, Maximum Entropy, Maximum Mutual Information, the Kullback-Leibler Information (Cross Entropy), Penalized Maximum Likelihood Estimation and Smoothing Splines. ::: The connection to Bayesian Inference is discussed. ::: Experimental results on the NIST OCR database, the Turk and Pentland human face database and sunspot time series prediction are presented which demonstrate that the ensemble method dramatically improves regression performance on real-world classification tasks.
---
paper_title: Optimal Linear Combinations Of Neural Networks

paper_content:

Neural network-based modeling often involves trying multiple networks with different architectures and training parameters in order to achieve acceptable model accuracy. Typically, one of the trained networks is chosen as best, while the rest are discarded. [Hashem and Schmeiser (1995)] proposed using optimal linear combinations of a number of trained neural networks instead of using a single best network. Combining the trained networks may help integrate the knowledge acquired by the components networks and thus improve model accuracy. In this paper, we extend the idea of optimal linear combinations (OLCs) of neural networks and discuss issues related to the generalization ability of the combined model. We then present two algorithms for selecting the component networks for the combination to improve the generalization ability of OLCs. Our experimental results demonstrate significant improvements in model accuracy, as a result of using OLCs, compared to using the apparent best network. Copyright 1997 Elsevier Science Ltd.
---
paper_title: The Combination of Forecasts

paper_content:

Two separate sets of forecasts of airline passenger data have been combined to form a composite set of forecasts. The main conclusion is that the composite set of forecasts can yield lower mean-square error than either of the original forecasts. Past errors of each of the original forecasts are used to determine the weights to attach to these two original forecasts in forming the combined forecasts, and different methods of deriving these weights are examined.
---
paper_title: Generalization error of ensemble estimators

paper_content:

It has been empirically shown that a better estimate with less generalization error can be obtained by averaging outputs of multiple estimators. This paper presents an analytical result for the generalization error of ensemble estimators. First, we derive a general expression of the ensemble generalization error by using factors of interest (bias, variance, covariance, and noise variance) and show how the generalization error is affected by each of them. Some special cases are then investigated. The result of a simulation is shown to verify our analytical result. A practically important problem of the ensemble approach, ensemble dilemma, is also discussed.
---
paper_title: Neural Networks and the Bias/Variance Dilemma

paper_content:

Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.
---
paper_title: Generalization error of ensemble estimators

paper_content:

It has been empirically shown that a better estimate with less generalization error can be obtained by averaging outputs of multiple estimators. This paper presents an analytical result for the generalization error of ensemble estimators. First, we derive a general expression of the ensemble generalization error by using factors of interest (bias, variance, covariance, and noise variance) and show how the generalization error is affected by each of them. Some special cases are then investigated. The result of a simulation is shown to verify our analytical result. A practically important problem of the ensemble approach, ensemble dilemma, is also discussed.
---
paper_title: Neural Networks and the Bias/Variance Dilemma

paper_content:

Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.
---
paper_title: The Combination of Forecasts

paper_content:

Two separate sets of forecasts of airline passenger data have been combined to form a composite set of forecasts. The main conclusion is that the composite set of forecasts can yield lower mean-square error than either of the original forecasts. Past errors of each of the original forecasts are used to determine the weights to attach to these two original forecasts in forming the combined forecasts, and different methods of deriving these weights are examined.
---
paper_title: Error Correlation and Error Reduction in Ensemble Classifiers

paper_content:

Using an ensemble of classifiers, instead of a single classifier, can lead to improved generalization. The gains obtained by combining, however, are often affected more by the selection of what is presented to the combiner than by the actual combining method that is chosen. In this paper, we focus on data selection and classifier training methods, in order to 'prepare' classifiers for combining. We review a combining framework for classification problems that quantifies the need for reducing the correlation among individual classifiers. Then, we discuss several methods that make the classifiers in an ensemble more complementary. Experimental results are provided to illustrate the benefits and pitfalls of reducing the correlation among classifiers, especially when the training data are in limited supply.
---
paper_title: Analysis of Linear and Order Statistics Combiners for Fusion of Imbalanced Classifiers

paper_content:

So far few theoretical works investigated the conditions under which specific fusion rules can work well, and a unifying framework for comparing rules of different complexity is clearly beyond the state of the art. A clear theoretical comparison is lacking even if one focuses on specific classes of combiners (e.g., linear combiners). In this paper, we theoretically compare simple and weighted averaging rules for fusion of imbalanced classifiers. Continuing the work reported in [10], we get a deeper knowledge of classifiers imbalance effects in linear combiners. In addition, we experimentally compare the performance of linear and order statistics combiners for ensembles with different degrees of classifiers imbalance.
---
paper_title: Linear Combiners for Classifier Fusion: Some Theoretical and Experimental Results

paper_content:

In this paper, we continue the theoretical and experimental analysis of two widely used combining rules, namely, the simple and weighted average of classifier outputs, that we started in previous works. We analyse and compare the conditions which affect the performance improvement achievable by weighted average over simple average, and over individual classifiers, under the assumption of unbiased and uncorrelated estimation errors. Although our theoretical results have been obtained under strict assumptions, the reported experiments show that they can be useful in real applications, for designing multiple classifier systems based on linear combiners.
---
paper_title: Error Correlation and Error Reduction in Ensemble Classifiers

paper_content:

Using an ensemble of classifiers, instead of a single classifier, can lead to improved generalization. The gains obtained by combining, however, are often affected more by the selection of what is presented to the combiner than by the actual combining method that is chosen. In this paper, we focus on data selection and classifier training methods, in order to 'prepare' classifiers for combining. We review a combining framework for classification problems that quantifies the need for reducing the correlation among individual classifiers. Then, we discuss several methods that make the classifiers in an ensemble more complementary. Experimental results are provided to illustrate the benefits and pitfalls of reducing the correlation among classifiers, especially when the training data are in limited supply.
---
paper_title: Analysis of Linear and Order Statistics Combiners for Fusion of Imbalanced Classifiers

paper_content:

So far few theoretical works investigated the conditions under which specific fusion rules can work well, and a unifying framework for comparing rules of different complexity is clearly beyond the state of the art. A clear theoretical comparison is lacking even if one focuses on specific classes of combiners (e.g., linear combiners). In this paper, we theoretically compare simple and weighted averaging rules for fusion of imbalanced classifiers. Continuing the work reported in [10], we get a deeper knowledge of classifiers imbalance effects in linear combiners. In addition, we experimentally compare the performance of linear and order statistics combiners for ensembles with different degrees of classifiers imbalance.
---
paper_title: Linear Combiners for Classifier Fusion: Some Theoretical and Experimental Results

paper_content:

In this paper, we continue the theoretical and experimental analysis of two widely used combining rules, namely, the simple and weighted average of classifier outputs, that we started in previous works. We analyse and compare the conditions which affect the performance improvement achievable by weighted average over simple average, and over individual classifiers, under the assumption of unbiased and uncorrelated estimation errors. Although our theoretical results have been obtained under strict assumptions, the reported experiments show that they can be useful in real applications, for designing multiple classifier systems based on linear combiners.
---
paper_title: Neural network ensembles

paper_content:

Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. >
---
paper_title: Using Diversity in Preparing Ensembles of Classifiers Based on Different Feature Seature Subsets to Minimize Generalization Error

paper_content:

It is well known that ensembles of predictors produce better accuracy than a single predictor provided there is diversity in the ensemble. This diversity manifests itself as disagreement or ambiguity among the ensemble members. In this paper we focus on ensembles of classifiers based on different feature subsets and we present a process for producing such ensembles that emphasizes diversity (ambiguity) in the ensemble members. This emphasis on diversity produces ensembles with low generalization errors from ensemble members with comparatively high generalization error. We compare this with ensembles produced focusing only on the error of the ensemble members (without regard to overall diversity) and find that the ensembles based on ambiguity have lower generalization error. Further, we find that the ensemble members produced focusing on ambiguity have less features on average that those based on error only. We suggest that this indicates that these ensemble members are local learners.
---
paper_title: Bagging and Boosting for the Nearest Mean Classifier: Effects of Sample Size on Diversity and Accuracy

paper_content:

In combining classifiers, it is believed that diverse ensembles perform better than non-diverse ones. In order to test this hypothesis, we study the accuracy and diversity of ensembles obtained in bagging and boosting applied to the nearest mean classifier. In our simulation study we consider two diversity measures: the Q statistic and the disagreement measure. The experiments, carried out on four data sets have shown that both diversity and the accuracy of the ensembles depend on the training sample size. With exception of very small training sample sizes, both bagging and boosting are more useful when ensembles consist of diverse classifiers. However, in boosting the relationship between diversity and the efficiency of ensembles is much stronger than in bagging.
---
paper_title: Diversity versus Quality in Classification Ensembles based on Feature Selection

paper_content:

Feature subset-selection has emerged as a useful technique for creating diversity in ensembles - particularly in classification ensembles. In this paper we argue that this diversity needs to be monitored in the creation of the ensemble. We propose an entropy measure of the outputs of the ensemble members as a useful measure of the ensemble diversity. Further, we show that using the associated conditional entropy as a loss function (error measure) works well and the entropy in the ensemble predicts well the reduction in error due to the ensemble. These measures are evaluated on a medical prediction problem and are shown to predict the performance of the ensemble well. We also show that the entropy measure of diversity has the added advantage that it seems to model the change in diversity with the size of the ensemble.
---
paper_title: Relationships between combination methods and measures of diversity in combining classifiers

paper_content:

Abstract This study looks at the relationships between different methods of classifier combination and different measures of diversity. We considered 10 combination methods and 10 measures of diversity on two benchmark data sets. The relationship was sought on ensembles of three classifiers built on all possible partitions of the respective feature sets into subsets of pre-specified sizes. The only positive finding was that the Double-Fault measure of diversity and the measure of difficulty both showed reasonable correlation with Majority Vote and Naive Bayes combinations. Since both these measures have an indirect connection to the ensemble accuracy, this result was not unexpected. However, our experiments did not detect a consistent relationship between the other measures of diversity and the 10 combination methods.
---
paper_title: Ten measures of diversity in classifier ensembles: limits for two classifiers

paper_content:

Independence and dependence of classifier outputs have been debated in the recent literature giving rise to notions such as diversity, complementarity, orthogonality, etc. There seems to be no consensus on the meaning of these notions beyond the intuitive perception. We summarize 10 measures of classifier diversity: 4 pairwise and 6 non-pairwise measures. We derive the limits of the measures for 2 classifiers of equal accuracy. (10 pages)
---
paper_title: Is independence good for combining classifiers?

paper_content:

Independence between individual classifiers is typically viewed as an asset in classifier fusion. We study the limits on the majority vote accuracy when combining dependent classifiers. Q statistics are used to measure the dependence between classifiers. We show that dependent classifiers could offer a dramatic improvement over the individual accuracy. However, the relationship between dependency and accuracy of the pool is ambivalent. A synthetic experiment demonstrates the intuitive result that, in general, negative dependence is preferable.
---
paper_title: Limits on the majority vote accuracy in classifier fusion

paper_content:

We derive upper and lower limits on the majority vote accuracy with respect to individual accuracy p, the number of classiers in the pool (L), and the pairwise dependence between classiers, measured by Yule's Q statistic. Independence between individual classi- ers is typically viewed as an asset in classier fusion. We show that the majority vote with dependent classiers can potentially oer a dramatic improvement both over independent clas- siers and over the individual accuracy p. A functional relationship between the limits and the pairwise dependence Q is derived. Two patterns of the joint distribution for classier outputs (correct/incorrect) are identied to derive the limits: the pattern of success and the pattern of failure. The results support the intuition that negative pairwise dependence is benecial although not straightforwardly related to the accuracy. The pattern of success showed that for the highest improvement over p, all pairs of classiers in the pool should have the same negative dependence.
---
paper_title: Bagging Predictors

paper_content:

Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.
---
paper_title: Ensemble Learning using Decorrelated Neural Networks

paper_content:

We describe a decorrelation network training method for improving the quality of regression learning in 'ensemble' neural networks NNs that are composed of linear combinations of individual NNs. In this method, individual networks are trained by backpropogation not only to reproduce a desired output, but also to have their errors linearly decorrelated with the other networks. Outputs from the individual networks are then linearly combined to produce the output of the ensemble network. We demonstrate the performances of decorrelated network training on learning the 'three-parity' logic function, a noisy sine function and a one-dimensional non-linear function, and compare the results with the ensemble networks composed of independently trained individual networks without decorrelation training . Empirical results show than when individual networks are forced to be decorrelated with one another the resulting ensemble NNs have lower mean squared errors than the ensemble networks having independently trained i...
---
paper_title: Popular Ensemble Methods: An Empirical Study

paper_content:

An ensemble consists of a set of individually trained classifiers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble is often more accurate than any of the single classifiers in the ensemble. Bagging (Breiman, 1996c) and Boosting (Freund & Schapire, 1996; Schapire, 1990) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods on 23 data sets using both neural networks and decision trees as our classification algorithm. Our results clearly indicate a number of conclusions. First, while Bagging is almost always more accurate than a single classifier, it is sometimes much less accurate than Boosting. On the other hand, Boosting can create ensembles that are less accurate than a single classifier - especially when using neural networks. Analysis indicates that the performance of the Boosting methods is dependent on the characteristics of the data set being examined. In fact, further results show that Boosting ensembles may overfit noisy data sets, thus decreasing its performance. Finally, consistent with previous studies, our work suggests that most of the gain in an ensemble's performance comes in the first few classifiers combined; however, relatively large gains can be seen up to 25 classifiers when Boosting decision trees.
---
paper_title: Fast committee learning: preliminary results

paper_content:

Fast committee learning can, to some extent, achieve the generalisation advantages of a committee of neural networks, without the need for independent learning of the committee members. This is achieved by selecting committee members from time-slices of the learning trajectory of one neural network.
---
paper_title: Combining the Predictions of Multiple Classifiers: Using Competitive Learning to Initialize Neural Networks

paper_content:

The primary goal of inductive learning is to generalize well - that is, induce a function that accurately produces the correct output for future inputs. Hansen and Salamon showed that, under certain assumptions, combining the predictions of several separately trained neural networks will improve generalization. One of their key assumptions is that the individual networks should be independent in the errors they produce. In the standard way of performing backpropagation this assumption may be violated, because the standard procedure is to initialize network weights in the region of weight space near the origin. This means that backpropagation's gradient-descent search may only reach a small subset of the possible local minima. In this paper we present an approach to initializing neural networks that uses competitive learning to intelligently create networks that are originally located far from the origin of weight space, thereby potentially increasing the set of reachable local minima. We report experiments on two real-world datasets where combinations of networks initialized with our method generalize better than combinations of networks initialized the traditional way.
---
paper_title: Improving Committee Diagnosis with Resampling Techniques

paper_content:

Central to the performance improvement of a committee relative to individual networks is the error correlation between networks in the committee. We investigated methods of achieving error independence between the networks by training the networks with different resampling sets from the original training set. The methods were tested on the sinwave artificial task and the real-world problems of hepatoma (liver cancer) and breast cancer diagnoses.
---
paper_title: Engineering Multiversion Neural-Net Systems

paper_content:

In this paper we address the problem of constructing reliable neural-net implementations, given the assumption that any particular implementation will not be totally correct. The approach taken in this paper is to organize the inevitable errors so as to minimize their impact in the context of a multiversion system, i.e., the system functionality is reproduced in multiple versions, which together will constitute the neural-net system. The unique characteristics of neural computing are exploited in order to engineer reliable systems in the form of diverse, multiversion systems that are used together with a "decision strategy" (such as majority vote). Theoretical notions of "methodological diversity" contributing to the improvement of system performance are implemented and tested. An important aspect of the engineering of an optimal system is to overproduce the components and then choose an optimal subset. Three general techniques for choosing final system components are implemented and evaluated. Several different approaches to the effective engineering of complex multiversion systems designs are realized and evaluated to determine overall reliability as well as reliability of the overall system in comparison to the lesser reliability of component substructures.
---
paper_title: The "test and Select" Approach to Ensemble Combination

paper_content:

The performance of neural nets can be improved through the use of ensembles of redundant nets. In this paper, some of the available methods of ensemble creation are reviewed and the "test and select" methodolology for ensemble creation is considered. This approach involves testing potential ensemble combinations on a validation set, and selecting the best performing ensemble on this basis, which is then tested on a final test set. The application of this methodology, and of ensembles in general, is explored further in two case studies. The first case study is of fault diagnosis in a diesel engine, and relies on ensembles of nets trained from three different data sources. The second case study is of robot localisation, using an evidence-shifting method based on the output of trained SOMs. In both studies, improved results are obtained as a result of combining nets to form ensembles.
---
paper_title: Experiments with Classifier Combining Rules

paper_content:

A large experiment on combining classifiers is reported and discussed. It includes, both, the combination of different classifiers on the same feature set and the combination of classifiers on different feature sets. Various fixed and trained combining rules are used. It is shown that there is no overall winning combining rule and that bad classifiers as well as bad feature sets may contain valuable information for performance improvement by combining rules. Best performance is achieved by combining both, different feature sets and different classifiers.
---
paper_title: Error-Correcting Output Codes: A General Method for Improving Multiclass Inductive Learning Programs

paper_content:

Multiclass learning problems involve finding a definition for an unknown function f(x) whose range is a discrete set containing k < 2 values (i.e., k "classes"). The definition is acquired by studying large collections of training examples of the form [xi, f(xi)]. Existing approaches to this problem include (a) direct application of multiclass algorithms such as the decision-tree algorithms ID3 and CART, (b) application of binary concept learning algorithms to learn individual binary functions for each of the k classes, and (c) application of binary concept learning algorithms with distributed output codes such as those employed by Sejnowski and Rosenberg in the NETtalk system. This paper compares these three approaches to a new technique in which BCH error-correcting codes are employed as a distributed output representation. We show that these output representations improve the performance of ID3 on the NETtalk task and of back propagation on an isolated-letter speech-recognition task. These results demonstrate that error-correcting output codes provide a general-purpose method for improving the performance of inductive learning programs on multiclass problems.
---
paper_title: Using Diversity in Preparing Ensembles of Classifiers Based on Different Feature Seature Subsets to Minimize Generalization Error

paper_content:

It is well known that ensembles of predictors produce better accuracy than a single predictor provided there is diversity in the ensemble. This diversity manifests itself as disagreement or ambiguity among the ensemble members. In this paper we focus on ensembles of classifiers based on different feature subsets and we present a process for producing such ensembles that emphasizes diversity (ambiguity) in the ensemble members. This emphasis on diversity produces ensembles with low generalization errors from ensemble members with comparatively high generalization error. We compare this with ensembles produced focusing only on the error of the ensemble members (without regard to overall diversity) and find that the ensembles based on ambiguity have lower generalization error. Further, we find that the ensemble members produced focusing on ambiguity have less features on average that those based on error only. We suggest that this indicates that these ensemble members are local learners.
---
paper_title: Boosting with averaged weight vectors

paper_content:

AdaBoost [5] is a well-known ensemble learning algorithm that constructs its constituent or base models in sequence. A key step in AdaBoost is constructing a distribution over the training examples to create each base model. This distribution, represented as a vector, is constructed to be orthogonal to the vector of mistakes made by the previous base model in the sequence [7]. The idea is to make the next base model's errors uncorrelated with those of the previous model. Some researchers have pointed out the intuition that it is probably better to construct a distribution orthogonal to the mistake vectors of all the previous base models, but that this is not always possible [7]. We present an algorithm that attempts to come as close as possible to this goal in an efficient manner. We present experimental results demonstrating significant improvement over AdaBoost and the Totally Corrective boosting algorithm [7], which also attempts to satisfy this goal.
---
paper_title: Diverse neural net solutions to a fault diagnosis problem

paper_content:

The development of a neural net system for fault diagnosis in a marine diesel engine is described. Nets were trained to classify combustion quality on the basis of simulated data. Three different types of data were used: pressure, temperature and combined pressure and temperature. Subsequent to training, three nets were selected and combined by means of a majority voter to form a system which achieved 100% generalisation to the test set. This performance is attributable to a reliance on the software engineering concept of diversity. Following experimental evaluation of methods of creating diverse neural nets solutions, it was concluded that the best results should be obtained when data is taken from two different sensors (e.g. a pressure and a temperature sensor), or where this is not possible, when new data sets are created by subjecting a set of inputs to non-linear transformations. These conclusions have far reaching implications for other neural net applications.
---
paper_title: Error-Correcting Output Coding Corrects Bias and Variance

paper_content:

Previous research has shown that a technique called error-correcting output coding (ECOC) can dramatically improve the classification accuracy of supervised learning algorithms that learn to classify data points into one of k ≫ 2 classes. This paper presents an investigation of why the ECOC technique works, particularly when employed with decision-tree learning algorithms. It shows that the ECOC method— like any form of voting or committee—can reduce the variance of the learning algorithm. Furthermore—unlike methods that simply combine multiple runs of the same learning algorithm—ECOC can correct for errors caused by the bias of the learning algorithm. Experiments show that this bias correction ability relies on the non-local behavior of C4.5.
---
paper_title: Bagging Predictors

paper_content:

Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.
---
paper_title: Randomizing Outputs to Increase Prediction Accuracy

paper_content:

Bagging and boosting reduce error by changing both the inputs and outputs to form perturbed training sets, growing predictors on these perturbed training sets and combining them. An interesting question is whether it is possible to get comparable performance by perturbing the outputs alone. Two methods of randomizing outputs are experimented with. One is called output smearing and the other output flipping. Both are shown to consistently do better than bagging.
---
paper_title: Input decimation ensembles: Decorrelation through dimensionality reduction

paper_content:

Using an ensemble of classifiers instead of a single classifier has been shown to improve generalization performance in many machine learning problems [4, 16]. However, the extent of such improvement depends greatly on the amount of correlation among the errors of the base classifiers [1,14]. As such, reducing those correlations while keeping the base classifiers' performance levels high is a promising research topic. In this paper, we describe input decimation, a method that decouples the base classifiers by training them with different subsets of the input features. In past work [15], we showed the theoretical benefits of input decimation and presented its application to a handful of real data sets. In this paper, we provide a systematic study of input decimation on synthetic data sets and analyze how the interaction between correlation and performance in base classifiers affects ensemble performance.
---
paper_title: Combining Decision Trees and Neural Networks for Drug Discovery

paper_content:

Genetic programming (GP) offers a generic method of automatically fusing together classifiers using their receiver operating characteristics (ROC) to yield superior ensembles. We combine decision trees (C4.5) and artificial neural networks (ANN) on a difficult pharmaceutical data mining (KDD) drug discovery application. Specifically predicting inhibition of a P450 enzyme. Training data came from high throughput screening (HTS) runs. The evolved model may be used to predict behaviour of virtual (i.e. yet to be manufactured) chemicals. Measures to reduce over fitting are also described.
---
paper_title: A Constructive Algorithm for Training Cooperative Neural Network Ensembles

paper_content:

Presents a constructive algorithm for training cooperative neural-network ensembles (CNNEs). CNNE combines ensemble architecture design with cooperative training for individual neural networks (NNs) in ensembles. Unlike most previous studies on training ensembles, CNNE puts emphasis on both accuracy and diversity among individual NNs in an ensemble. In order to maintain accuracy among individual NNs, the number of hidden nodes in individual NNs are also determined by a constructive approach. Incremental training based on negative correlation is used in CNNE to train individual NNs for different numbers of training epochs. The use of negative correlation learning and different training epochs for training individual NNs reflect CNNEs emphasis on diversity among individual NNs in an ensemble. CNNE has been tested extensively on a number of benchmark problems in machine learning and neural networks, including Australian credit card assessment, breast cancer, diabetes, glass, heart disease, letter recognition, soybean, and Mackey-Glass time series prediction problems. The experimental results show that CNNE can produce NN ensembles with good generalization ability.
---
paper_title: Network generalization differences quantified

paper_content:

Abstract It has long been observed, and frequently noted, by connectionists that small changes in initial conditions, prior to training, can result in networks that generalize very differently. We have performed a systematic study of this phenomenon, using a number of different statistical measures of generalization differences. From these we derive a formal definition of Generalization Diversity. We quantify the relative impacts on generalization of the major parameters used in network initialization as well as extend the formal framework to also encompass the differences in generalization difference from one parameter to another. We reveal, for example, the relative effects of random initialization of the link weights and variation of the number of hidden units, and how similar these two resultant effects are. Finally, examples are presented of how the proposed generalization diversity measure may be exploited in order to improve the performance of neural-net systems. We show how several of these measures can be used to engineer reliability improvements in neural-net systems.
---
paper_title: Generating Accurate and Diverse Members of a Neural-Network Ensemble

paper_content:

Neural-network ensembles have been shown to be very accurate classification techniques. Previous work has shown that an effective ensemble should consist of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well. Most existing techniques, however, only indirectly address the problem of creating such a set of networks. In this paper we present a technique called ADDEMUP that uses genetic algorithms to directly search for an accurate and diverse set of trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to continually create new networks, keeping the set of networks that are as accurate as possible while disagreeing with each other as much as possible. Experiments on three DNA problems show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing approaches. Experiments also show that ADDEMUP is able to effectively incorporate prior knowledge, if available, to improve the quality of its ensemble.
---
paper_title: Diversity between Neural Networks and Decision Trees for Building Multiple Classifier Systems

paper_content:

A multiple classifier system can only improve the performance when the members in the system are diverse from each other. Combining some methodologically different techniques is considered a constructive way to expand the diversity. This paper investigates the diversity between the two different data mining techniques, neural networks and automatically induced decision trees. Input decimation through salient feature selection is also explored in the paper in the hope of acquiring further diversity. Among various diversities defined, the coincident failure diversity (CFD) appears to be an effective measure of useful diversity among classifiers in a multiple classifier system when the majority voting decision strategy is applied. A real-world medical classification problem is presented as an application of the techniques. The constructed multiple classifier systems are evaluated with a number of statistical measures in terms of reliability and generalisation. The results indicate that combined MCSs of the nets and trees trained with the selected features have higher diversity and produce better classification results.
---
paper_title: Engineering Multiversion Neural-Net Systems

paper_content:

In this paper we address the problem of constructing reliable neural-net implementations, given the assumption that any particular implementation will not be totally correct. The approach taken in this paper is to organize the inevitable errors so as to minimize their impact in the context of a multiversion system, i.e., the system functionality is reproduced in multiple versions, which together will constitute the neural-net system. The unique characteristics of neural computing are exploited in order to engineer reliable systems in the form of diverse, multiversion systems that are used together with a "decision strategy" (such as majority vote). Theoretical notions of "methodological diversity" contributing to the improvement of system performance are implemented and tested. An important aspect of the engineering of an optimal system is to overproduce the components and then choose an optimal subset. Three general techniques for choosing final system components are implemented and evaluated. Several different approaches to the effective engineering of complex multiversion systems designs are realized and evaluated to determine overall reliability as well as reliability of the overall system in comparison to the lesser reliability of component substructures.
---
paper_title: Combination of multiple classifiers using local accuracy estimates

paper_content:

This paper presents a method for combining classifiers that uses estimates of each individual classifier's local accuracy in small regions of feature space surrounding an unknown test sample. An empirical evaluation using five real data sets confirms the validity of our approach compared to some other combination of multiple classifiers algorithms. We also suggest a methodology for determining the best mix of individual classifiers.
---
paper_title: Modelling conditional probabilities with network committees: how overfitting can be useful

paper_content:

Training neural networks for predicting conditional probability densities can be accelerated considerably by adopting the random vector functional link net (RVFL) approach. ::: In this way, a whole ensemble of models can be trained at the same computational ::: costs as otherwise required for training only one conventional network. The inherent ::: stochasticity of the RVFL method increases the diversity in this ensemble, which leads ::: to a signi cant reduction of the generalisation error. The application of this scheme to ::: a synthetic multimodal stochastic time series and a real-world benchmark problem was ::: found to achieve a performance better than or comparable to the best results otherwise obtained so far. Moreover, the simulations support a recent theoretical study and ::: show that when making predictions with network committees, it can be advantageous ::: to employ underregularised models that overfit the training data
---
paper_title: Ensemble Learning via Negative Correlation

paper_content:

This paper presents a learning approach, i.e. negative correlation learning, for neural network ensembles. Unlike previous learning approaches for neural network ensembles, negative correlation learning attempts to train individual networks in an ensemble and combines them in the same learning process. In negative correlation learning, all the individual networks in the ensemble are trained simultaneously and interactively through the correlation penalty terms in their error functions. Rather than producing unbiased individual networks whose errors are uncorrelated, negative correlation learning can create negatively correlated networks to encourage specialisation and cooperation among the individual networks. Empirical studies have been carried out to show why and how negative correlation learning works. The experimental results show that negative correlation learning can produce neural network ensembles with good generalisation ability.
---
paper_title: The Use of the Ambiguity Decomposition in Neural Network Ensemble Learning Methods

paper_content:

We analyze the formal grounding behind Negative Correlation (NC) Learning, an ensemble learning technique developed in the evolutionary computation literature. We show that by removing an assumption made in the original work, NC can be seen to be exploiting the well-known Ambiguity decomposition of the ensemble error, grounding it in a statistics framework around the bias-variance decomposition. We use this grounding to find bounds for the parameters, and provide insights into the behaviour of the optimal parameter values. These observations allow us understand how NC relates to other algorithms, identifying a group of papers spread over the last decade that have all exploited the Ambiguity decomposition for machine learning problems. When taking into account our new understanding of the algorithm, significant reductions in error rates were observed in empirical tests.
---
paper_title: Learning with ensembles: How overfitting can be useful

paper_content:

We study the characteristics of learning with ensembles. Solving exactly the simple model of an ensemble of linear students, we find surprisingly rich behaviour. For learning in large ensembles, it is advantageous to use under-regularized students, which actually over-fit the training data. Globally optimal performance can be obtained by choosing the training set sizes of the students appropriately. For smaller ensembles, optimization of the ensemble weights can yield significant improvements in ensemble generalization performance, in particular if the individual students are subject to noise in the training process. Choosing students with a wide range of regularization parameters makes this improvement robust against changes in the unknown level of noise in the training data.
---
paper_title: Ensemble Learning using Decorrelated Neural Networks

paper_content:

We describe a decorrelation network training method for improving the quality of regression learning in 'ensemble' neural networks NNs that are composed of linear combinations of individual NNs. In this method, individual networks are trained by backpropogation not only to reproduce a desired output, but also to have their errors linearly decorrelated with the other networks. Outputs from the individual networks are then linearly combined to produce the output of the ensemble network. We demonstrate the performances of decorrelated network training on learning the 'three-parity' logic function, a noisy sine function and a one-dimensional non-linear function, and compare the results with the ensemble networks composed of independently trained individual networks without decorrelation training . Empirical results show than when individual networks are forced to be decorrelated with one another the resulting ensemble NNs have lower mean squared errors than the ensemble networks having independently trained i...
---
paper_title: Genetic Programming: On the Programming of Computers by Means of Natural Selection

paper_content:

Background on genetic algorithms, LISP, and genetic programming hierarchical problem-solving introduction to automatically-defined functions - the two-boxes problem problems that straddle the breakeven point for computational effort Boolean parity functions determining the architecture of the program the lawnmower problem the bumblebee problem the increasing benefits of ADFs as problems are scaled up finding an impulse response function artificial ant on the San Mateo trail obstacle-avoiding robot the minesweeper problem automatic discovery of detectors for letter recognition flushes and four-of-a-kinds in a pinochle deck introduction to biochemistry and molecular biology prediction of transmembrane domains in proteins prediction of omega loops in proteins lookahead version of the transmembrane problem evolutionary selection of the architecture of the program evolution of primitives and sufficiency evolutionary selection of terminals evolution of closure simultaneous evolution of architecture, primitive functions, terminals, sufficiency, and closure the role of representation and the lens effect. Appendices: list of special symbols list of special functions list of type fonts default parameters computer implementation annotated bibliography of genetic programming electronic mailing list and public repository.
---
paper_title: Speciated neural networks evolved with fitness sharing technique

paper_content:

In order to develop effective evolutionary artificial neural networks (EANNs) we have to address the questions on how to evolve EANNs more efficiently and how to achieve the best performance from the ANNs evolved. Most of the previous works, however, do not utilize all the information obtained with several ANNs but choose the one best network in the last generation. Some recent works indicate that making use of population information by combining ANNs in the last generation can improve the performance, because they can complement each other to construct effective multiple neural networks. We propose a new method of evolving multiple speciated neural networks by fitness sharing which helps to optimize multi-objective functions with genetic algorithms. Experiments with the breast cancer data from UCI benchmark datasets show that the proposed method can produce more speciated ANNs and improve the performance by combining the only representative individuals.
---
paper_title: Speciation as automatic categorical modularization

paper_content:

Many natural and artificial systems use a modular approach to reduce the complexity of a set of subtasks while solving the overall problem satisfactorily. There are two distinct ways to do this. In functional modularization, the components perform very different tasks, such as subroutines of a large software project. In categorical modularization, the components perform different versions of basically the same task, such as antibodies in the immune system. This second aspect is the more natural for acquiring strategies in games of conflict, An evolutionary learning system is presented which follows this second approach to automatically create a repertoire of specialist strategies for a game-playing system. This relieves the human effort of deciding how to divide and specialize. The genetic algorithm speciation method used is one based on fitness sharing. The learning task is to play the iterated prisoner's dilemma. The learning system outperforms the tit-for-tat strategy against unseen test opponents. It learns using a "black box" simulation, with minimal prior knowledge of the learning task.
---
paper_title: Making Use of Population Information in Evolutionary Artificial Neural Networks

paper_content:

This paper is concerned with the simultaneous evolution of artificial neural network (ANN) architectures and weights. The current practice in evolving ANN's is to choose the best ANN in the last generation as the final result. This paper proposes a different approach to form the final result by combining all the individuals in the last generation in order to make best use of all the information contained in the whole population. This approach regards a population of ANN's as an ensemble and uses a combination method to integrate them. Although there has been some work on integrating ANN modules, little has been done in evolutionary learning to make best use of its population information. Four linear combination methods have been investigated in this paper to illustrate our ideas. Three real-world data sets have been used in our experimental studies, which show that the recursive least-square (RLS) algorithm always produces an integrated system that outperforms the best individual. The results confirm that a population contains more information than a single individual. Evolutionary learning should exploit such information to improve generalization of learned systems.
---
paper_title: Artificial speciation of neural network ensembles

paper_content:

Modular approach of solving a complex problem can reduce the total complexity of the system while solving a difficult problem satisfactorily. To implement this idea, an EANN system is developed here for classifying data. The system evolved is speciated in such a manner that members of a particular species solve certain parts of the problem and complement each other in solving one big problem. Fitness sharing is used in evolving the group of ANNs to achieve the required speciation. Sharing was performed at phenotypic level using modified Kullback-Leibler entropy as the distance measure. Since the group as a unit solves the classification problem, outputs of all the ANNs are used in finding the final output. For the combination of ANN outputs 3 different methods – Voting, averaging and recursive least square are used. The evolved system is tested on two data classification problems (Heart Disease Dataset and Breast Cancer Dataset) taken from UCI machine learning benchmark repository.
---
paper_title: Feature Selection for Ensembles

paper_content:

The traditional motivation behind feature selection algorithms is to find the best subset of features for a task using one particular learning algonthm. Given the recent success of ensembles, however, we investigate the notion of ensemble feature selection in this paper. This task is harder than traditional feature selection in that one not only needs to find features germane to the learning task and learning algorithm, but one also needs to find a set of feature subsets that will promote disagreement among the ensemble's classifiers. In this paper, we present an ensemble feature selection approach that is based on genetic algorithms. Our algorithm shows improved performance over the popular and powerful ensemble approaches of AdaBoost and Bagging and demonstrates the utility of ensemble feature selection.
---
paper_title: Types of Multinet System

paper_content:

A limiting factor in research on combining classifiers is a lack of awareness of the full range of available modular structures. One reason for this is that there is as yet little agreement on a means of describing and classifying types of multiple classifier system. In this paper, a categorisation scheme for the identification and description of types of multinet systems is proposed in which systems are described as (a) involving competitive or cooperative combination mechanisms; (b) combining either ensemble, modular, or hybrid components; (c) relying on either bottom-up, or top-down combination, and (d) when bottom up as using either static or fixed combination methods. It is claimed that the categorisation provides an early, but necessary, step in the process of mapping the space of multinet systems: permitting the comparison of different types of system, and facilitating their design and description. On the basis of this scheme, one ensemble and two modular multinet system designs are implemented, and applied to an engine fault diagnosis problem. The best generalisation performance was achieved from the ensemble system.
---
paper_title: Data Complexity Analysis for Classifier Combination

paper_content:

Multiple classifier methods are effective solutions to difficult pattern recognition problems. However, empirical successes and failures have not been completely explained. Amid the excitement and confusion, uncertainty persists in the optimality of method choices for specific problems due to strong data dependences of classifier performance. In response to this, I propose that further exploration of the methodology be guided by detailed descriptions of geometrical characteristics of data and classifier models.
---
paper_title: The Use of the Ambiguity Decomposition in Neural Network Ensemble Learning Methods

paper_content:

We analyze the formal grounding behind Negative Correlation (NC) Learning, an ensemble learning technique developed in the evolutionary computation literature. We show that by removing an assumption made in the original work, NC can be seen to be exploiting the well-known Ambiguity decomposition of the ensemble error, grounding it in a statistics framework around the bias-variance decomposition. We use this grounding to find bounds for the parameters, and provide insights into the behaviour of the optimal parameter values. These observations allow us understand how NC relates to other algorithms, identifying a group of papers spread over the last decade that have all exploited the Ambiguity decomposition for machine learning problems. When taking into account our new understanding of the algorithm, significant reductions in error rates were observed in empirical tests.
---
paper_title: Generating Accurate and Diverse Members of a Neural-Network Ensemble

paper_content:

Neural-network ensembles have been shown to be very accurate classification techniques. Previous work has shown that an effective ensemble should consist of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well. Most existing techniques, however, only indirectly address the problem of creating such a set of networks. In this paper we present a technique called ADDEMUP that uses genetic algorithms to directly search for an accurate and diverse set of trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to continually create new networks, keeping the set of networks that are as accurate as possible while disagreeing with each other as much as possible. Experiments on three DNA problems show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing approaches. Experiments also show that ADDEMUP is able to effectively incorporate prior knowledge, if available, to improve the quality of its ensemble.
---
paper_title: Error-Correcting Output Coding Corrects Bias and Variance

paper_content:

Previous research has shown that a technique called error-correcting output coding (ECOC) can dramatically improve the classification accuracy of supervised learning algorithms that learn to classify data points into one of k ≫ 2 classes. This paper presents an investigation of why the ECOC technique works, particularly when employed with decision-tree learning algorithms. It shows that the ECOC method— like any form of voting or committee—can reduce the variance of the learning algorithm. Furthermore—unlike methods that simply combine multiple runs of the same learning algorithm—ECOC can correct for errors caused by the bias of the learning algorithm. Experiments show that this bias correction ability relies on the non-local behavior of C4.5.
---
paper_title: Variance and Bias for General Loss Functions

paper_content:

When using squared error loss, bias and variance and their decomposition of prediction error are well understood and widely used concepts. However, there is no universally accepted definition for other loss functions. Numerous attempts have been made to extend these concepts beyond squared error loss. Most approaches have focused solely on 0-1 loss functions and have produced significantly different definitions. These differences stem from disagreement as to the essential characteristics that variance and bias should display. This paper suggests an explicit list of rules that we feel any “reasonable” set of definitions should satisfy. Using this framework, bias and variance definitions are produced which generalize to any symmetric loss function. We illustrate these statistics on several loss functions with particular emphasis on 0-1 loss. We conclude with a discussion of the various definitions that have been proposed in the past as well as a method for estimating these quantities on real data sets.
---
paper_title: Generalization error of ensemble estimators

paper_content:

It has been empirically shown that a better estimate with less generalization error can be obtained by averaging outputs of multiple estimators. This paper presents an analytical result for the generalization error of ensemble estimators. First, we derive a general expression of the ensemble generalization error by using factors of interest (bias, variance, covariance, and noise variance) and show how the generalization error is affected by each of them. Some special cases are then investigated. The result of a simulation is shown to verify our analytical result. A practically important problem of the ensemble approach, ensemble dilemma, is also discussed.
---
paper_title: Variance and Bias for General Loss Functions

paper_content:

When using squared error loss, bias and variance and their decomposition of prediction error are well understood and widely used concepts. However, there is no universally accepted definition for other loss functions. Numerous attempts have been made to extend these concepts beyond squared error loss. Most approaches have focused solely on 0-1 loss functions and have produced significantly different definitions. These differences stem from disagreement as to the essential characteristics that variance and bias should display. This paper suggests an explicit list of rules that we feel any “reasonable” set of definitions should satisfy. Using this framework, bias and variance definitions are produced which generalize to any symmetric loss function. We illustrate these statistics on several loss functions with particular emphasis on 0-1 loss. We conclude with a discussion of the various definitions that have been proposed in the past as well as a method for estimating these quantities on real data sets.
---

Title: Diversity Creation Methods: A Survey and Categorisation

Section 1: Introduction
Description 1: Provide an overview of the importance of creating diverse sets of classifiers, the challenges involved, and a summary of what this paper aims to achieve.

Section 2: When is an Ensemble Better than a Single Learner?
Description 2: Discuss why ensembles with diverse errors tend to perform better than single learners, covering both regression and classification contexts.

Section 3: In a Regression Context
Description 3: Present the historical context and foundational studies on combining regression estimators, and describe the mathematical frameworks used to understand diversity in regression.

Section 4: The Ambiguity Decomposition
Description 4: Explain the Ambiguity Decomposition, its significance for ensemble research, and how it provides a simple expression for error correlation effects in an ensemble.

Section 5: Bias, Variance and Covariance
Description 5: Discuss the Bias-Variance-Covariance decomposition, detailing the components of bias, variance, and covariance, and their effect on ensemble error.

Section 6: The Connection Between Ambiguity and Covariance
Description 6: Illustrate the mathematical and conceptual link between the Ambiguity Decomposition and the Bias-Variance-Covariance decomposition.

Section 7: In a Classification Context
Description 7: Explore the challenges and subtleties of quantifying diversity in classification problems, comparing it to the regression context.

Section 8: Ordinal Outputs
Description 8: Review theoretical work by Tumer and Ghosh on averaging combination rules and posterior probability estimates for classification ensembles.

Section 9: Non-Ordinal Outputs
Description 9: Examine methods and theoretical results for handling non-ordinal classifier outputs, covering topics like majority voting and classification error diversity.

Section 10: A Heuristic Metric for Classification Error Diversity?
Description 10: Summarize heuristic metrics and empirical studies aimed at measuring classification error diversity, including Sharkey's levels of diversity and Kuncheva's metrics.

Section 11: Towards A Taxonomy of Methods for Creating Diversity
Description 11: Propose a new taxonomy for categorising diversity creation methods in ensembles, split into explicit and implicit methods, and further divided into starting points, accessible hypotheses, and traversal methods in hypothesis space.

Section 12: How to Categorise Multiple Classifier Systems?
Description 12: Discuss different approaches to categorising multiple classifier systems, emphasizing the importance of a clear representation of the state of the field.

Section 13: How to Quantify Classification Diversity?
Description 13: Explore potential pathways to defining a bias-variance-covariance decomposition for zero-one loss functions to better understand classification error diversity.

Section 14: Conclusions
Description 14: Summarize the findings of the survey, highlighting the contribution of the categorisation of diversity creation methods and suggesting directions for future research.