A survey of multiple classifier systems as hybrid systems
---
paper_title: Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions

paper_content:

Ensemble methods have been called the most influential development in Data Mining and Machine Learning in the past decade. They combine multiple models into one usually more accurate than the best of its components. Ensembles can provide a critical boost to industrial challenges -- from investment timing to drug discovery, and fraud detection to recommendation systems -- where predictive accuracy is more vital than model interpretability. Ensembles are useful with all modeling algorithms, but this book focuses on decision trees to explain them most clearly. After describing trees and their strengths and weaknesses, the authors provide an overview of regularization -- today understood to be a key reason for the superior performance of modern ensembling algorithms. The book continues with a clear description of two recent developments: Importance Sampling (IS) and Rule Ensembles (RE). IS reveals classic ensemble methods -- bagging, random forests, and boosting -- to be special cases of a single algorithm, thereby showing how to improve their accuracy and speed. REs are linear rule models derived from decision tree ensembles. They are the most interpretable version of ensembles, which is essential to applications such as credit scoring and fault diagnosis. Lastly, the authors explain the paradox of how ensembles achieve greater accuracy on new data despite their (apparently much greater) complexity.This book is aimed at novice and advanced analytic researchers and practitioners -- especially in Engineering, Statistics, and Computer Science. Those with little exposure to ensembles will learn why and how to employ this breakthrough method, and advanced practitioners will gain insight into building even more powerful models. Throughout, snippets of code in R are provided to illustrate the algorithms described and to encourage the reader to try the techniques. (edited by author)
---
paper_title: Anycasting in connection-oriented computer networks: Models, algorithms and results

paper_content:

Our discussion in this article centers around various issues related to the use of anycasting in connection-oriented computer networks. Anycast is defined as a one-to-one-of-many transmission to deliver a packet to one of many hosts. Anycasting can be applied if the same content is replicated over many locations in the network. Examples of network techniques that apply anycasting are Content Delivery Networks (CDNs), Domain Name Service (DNS), Peer-to-Peer (P2P) systems. The role of anycasting is growing concurrently with the popularity of electronic music, movies, and other content required by Internet users. In this work we focus on the optimization of anycast flows in connection-oriented networks. We formulate a model of anycast connections and next propose a heuristic algorithm based on the Lagrangean relaxation aimed to optimize jointly routes for anycast and unicast connections. Results of numerical experiments are presented and evaluated. Finally, we analyze briefly problems related to anycasting in dynamic routing and multi-layer networks.
---
paper_title: Learning Decision Lists

paper_content:

This paper introduces a new representation for Boolean functions, called decision lists, and shows that they are efficiently learnable from examples. More precisely, this result is established for k-;DL – the set of decision lists with conjunctive clauses of size k at each decision. Since k-DL properly includes other well-known techniques for representing Boolean functions such as k-CNF (formulae in conjunctive normal form with at most k literals per clause), k-DNF (formulae in disjunctive normal form with at most k literals per term), and decision trees of depth k, our result strictly increases the set of functions that are known to be polynomially learnable, in the sense of Valiant (1984). Our proof is constructive: we present an algorithm that can efficiently construct an element of k-DL consistent with a given set of examples, if one exists.
---
paper_title: Optimizing group judgmental accuracy in the presence of interdependencies

paper_content:

Consider a group of people confronted with a dichotomous choice (for example, a yes or no decision). Assume that we can characterize each person by a probability, pi, of making the 'better' of the two choices open to the group, such that we define 'better' in terms of some linear ordering of the alternatives. If individual choices are independent, and if the a priori likelihood that either of the two choices is correct is one half, we show that the group decision procedure that maximizes the likelihood that the group will make the better of the two choices open to it is a weighted voting rule that assigns weights, wi, such that
---
paper_title: Fusion Methods for Unsupervised Learning Ensembles

paper_content:

The application of a “committee of experts” or ensemble learning to artificial neural networks that apply unsupervised learning techniques is widely considered to enhance the effectiveness of such networks greatly. This book examines in one of its chapters the potential of the ensemble meta-algorithm by describing and testing a technique based on the combination of ensembles and statistical PCA that is able to determine the presence of outliers in high-dimensional data sets and to minimize outlier effects in the final results. After that, it presents its central contribution, which consists on an algorithm for the ensemble fusion of topology-preserving maps, referred to as Weighted Voting Superposition (WeVoS), which has been devised to improve data exploration by 2-D visualization over multi-dimensional data sets. This generic algorithm is applied in combination with several other models taken from the family of topology preserving maps, such as the SOM, ViSOM, SIM and Max-SIM. A range of quality measures for topology preserving maps that are proposed in the literature are used to validate and compare WeVoS with other algorithms. The experimental results demonstrate that, in the majority of cases, the WeVoS algorithm outperforms earlier map-fusion methods and the simpler versions of the algorithm with which it is compared. All the algorithms are tested in different artificial data sets and in several of the most common machine-learning data sets in order to corroborate their theoretical properties. Moreover, a real-life case-study taken from the food industry demonstrates the practical benefits of their applications to more complex problems.
---
paper_title: Classifier Ensembles: Select Real-World Applications

paper_content:

Broad classes of statistical classification algorithms have been developed and applied successfully to a wide range of real-world domains. In general, ensuring that the particular classification algorithm matches the properties of the data is crucial in providing results that meet the needs of the particular application domain. One way in which the impact of this algorithm/application match can be alleviated is by using ensembles of classifiers, where a variety of classifiers (either different types of classifiers or different instantiations of the same classifier) are pooled before a final classification decision is made. Intuitively, classifier ensembles allow the different needs of a difficult problem to be handled by classifiers suited to those particular needs. Mathematically, classifier ensembles provide an extra degree of freedom in the classical bias/variance tradeoff, allowing solutions that would be difficult (if not impossible) to reach with only a single classifier. Because of these advantages, classifier ensembles have been applied to many difficult real-world problems. In this paper, we survey select applications of ensemble methods to problems that have historically been most representative of the difficulties in classification. In particular, we survey applications of ensemble methods to remote sensing, person recognition, one vs. all recognition, and medicine.
---
paper_title: Complexity and multithreaded implementation analysis of one class-classifiers fuzzy combiner

paper_content:

More recently, neural network techniques and fuzzy logic inference systems have been receiving an increasing attention. At the same time, methods of establishing decision by a group of classifiers are regarded as a general problem in various application areas of pattern recognition. Similarly to standard two-class pattern recognition methods, one-class classifiers hardly ever fit the data distribution perfectly. The paper presents fuzzy models for one-class classifier combination, compares their computational and expected space complexity with the one from ECOC and decision templates, presents possibility to speed up a fuser processing by means of multithreading.
---
paper_title: The CN2 induction algorithm

paper_content:

Systems for inducing concept descriptions from examples are valuable tools for assisting in the task of knowledge acquisition for expert systems. This paper presents a description and empirical evaluation of a new induction system. CN2, designed for the efficient induction of simple, comprehensible production rules in domains where problems of poor description language and/or noise may be present. Implementations of the CN2, ID3, and AQ algorithms are compared on three medical classification tasks.
---
paper_title: Queries and Concept Learning

paper_content:

We consider the problem of using queries to learn an unknown concept. Several types of queries are described and studied: membership, equivalence, subset, superset, disjointness, and exhaustiveness queries. Examples are given of efficient learning methods using various subsets of these queries for formal domains, including the regular languages, restricted classes of context-free languages, the pattern languages, and restricted types of prepositional formulas. Some general lower bound techniques are given. Equivalence queries are compared with Valiant's criterion of probably approximately correct identification under random sampling.
---
paper_title: Neural network ensembles

paper_content:

Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. >
---
paper_title: INTRODUCTION TO MACHINE LEARNINGIntroduction to Machine Learning

paper_content:

This chapter introduces the foundations of machine learning that are necessary to motivate its combination to evolutionary algorithms and image processing. In the chapter, we will describe several machine learning tools, suitable for a wide variety of data and tasks. The goal is not to describe the total number of techniques, but instead to present a general view of the field through the description of the mail problems considered in machine learning.
---
paper_title: An Introduction to Computational Learning Theory

paper_content:

The probably approximately correct learning model Occam's razor the Vapnik-Chervonenkis dimension weak and strong learning learning in the presence of noise inherent unpredictability reducibility in PAC learning learning finite automata by experimentation appendix - some tools for probabilistic analysis.
---
paper_title: The supervised learning no-free-lunch Theorems

paper_content:

This paper reviews the supervised learning versions of the no-free-lunch theorems in a simplified form. It also discusses the significance of those theorems, and their relation to other aspects of supervised learning.
---
paper_title: Bagging Predictors

paper_content:

Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.
---
paper_title: Ensemble Methods in Machine Learning

paper_content:

Ensemble methods are learning algorithms that construct a set of classifiers and then classify new data points by taking a (weighted) vote of their predictions. The original ensemble method is Bayesian averaging, but more recent algorithms include error-correcting output coding, Bagging, and boosting. This paper reviews these methods and explains why ensembles can often perform better than any single classifier. Some previous studies comparing ensemble methods are reviewed, and some new experiments are presented to uncover the reasons that Adaboost does not overfit rapidly.
---
paper_title: A composite classifier system design: Concepts and methodology

paper_content:

This study explores the scope for achieving enhanced recognition system performance through deployment of a composite classifier system consisting of two or more component classifiers which belong to different categories. The domains of deployment of these individual components (classifiers) are determined by optimal partitioning of the problem space. The criterion for such optimal partitioning is determined in each case by the characteristics of the classifier components. An example, in terms of partitioning the feature space for optimal deployment of a composite system consisting of the linear and nearest neighbor (NN) classifiers as its components, is presented to illustrate the concepts, the associated methodology, and the possible benefits one could expect through such composite classifier system design. Here, the optimality of the partitioning is dictated by the linear class separability limitation of the linear classifier and the computational demand characteristics of the NN classifier. Accordingly, the criterion for the optimal feature space partitioning is set to be the minimization of the domain of application of the NN classifier, subject to the constraint that the linear classifier is to be deployed only in regions satisfying the underlying assumption of linear separability of classes. While many alternatives are available for the solution of the resulting constrained optimization problem, a specific technique-Sequential Weight Increasing Factor Technique (SWIFT)- was employed here for convenience in view of previous successful experience with this technique in other application areas. Numerical results derived using the well-known IRIS data set are furnished to demonstrate the effectiveness of the new concepts and methodology.
---
paper_title: Privacy-preserving data mining

paper_content:

A fruitful direction for future data mining research will be the development of techniques that incorporate privacy concerns. Specifically, we address the following question. Since the primary task in data mining is the development of models about aggregated data, can we develop accurate models without access to precise information in individual data records? We consider the concrete case of building a decision-tree classifier from training data in which the values of individual records have been perturbed. The resulting data records look very different from the original records and the distribution of data values is also very different from the original distribution. While it is not possible to accurately estimate original values in individual data records, we propose a novel reconstruction procedure to accurately estimate the distribution of original data values. By using these reconstructed distributions, we are able to build classifiers whose accuracy is comparable to the accuracy of classifiers built with the original data.
---
paper_title: Design of effective multiple classifier systems by clustering of classifiers

paper_content:

In the field of pattern recognition, multiple classifier systems based on the combination of outputs of a set of different classifiers have been proposed as a method for the development of high performance classification systems. Previous work clearly showed that multiple classifier. Systems are effective only if the classifiers forming them make independent errors. Therefore, the fundamental need for methods aimed to design "error-independent" classifiers is currently acknowledged. In the paper, an approach to the automatic design of multiple classifier systems is proposed. Given an initial large set of classifiers, our approach is aimed at selecting the subset formed by the most error-independent classifiers. Reported results on the classification of multisensor remote-sensing images show that this approach allows to design effective multiple classifier systems.
---
paper_title: A Two-Stage Classifier with Reject Option for Text Categorisation

paper_content:

In this paper, we investigate the usefulness of the reject option in text categorisation systems. The reject option is introduced by allowing a text classifier to withhold the decision of assigning or not a document to any subset of categories, for which the decision is considered not sufficiently reliable. To automatically handle rejections, a two-stage classifier architecture is used, in which documents rejected at the first stage are automatically classified at the second stage, so that no rejections eventually remain. The performance improvement achievable by using the reject option is assessed on a real text categorisation task, using the well known Reuters data set.
---
paper_title: Serial Combination of Multiple Experts: A Unified Evaluation

paper_content:

Multiple expert decision combination has received much attention in recent years. This is a multi-disciplinary branch of pattern recognition which has extensive applications in numerous fields including robotic vision, artificial intelligence, document processing, office automation, human-computer interfaces, data acquisition, storage and retrieval, etc. In recent years, this application area has been extended to forensic science, including the identification of individuals using measures depending on biometrics, security and other applications. In this paper, a generalised multi-expert multi-level decision combination strategy, the serial combination approach, has been investigated from the dual viewpoints of theoretical analysis and practical implementation. Different researchers have implicitly utilised various approaches based on this concept over the years in a wide spectrum of application domains, but a comprehensive, coherent and generalised presentation of this approach from both theoretical and implementation viewpoints has not been attempted. While presenting here a unified framework for serial multiple expert decision combination, it is shown that many multi-expert approaches reported in the literature can be easily represented within the proposed framework. Detailed theoretical and practical discussions of the various performance results with these combinations, analysis of the internal processing of this approach, a case study for testing the theoretical framework, issues relating to processing overheads associated with the implementation of this approach, general comments on its applicability to various task domains and the generality of the approach in terms of reevaluating previous research have also been incorporated.
---
paper_title: Fusion Of Face Recognition Algorithms For Video-Based Surveillance Systems

paper_content:

It is widely acknowledged that face recognition could play an important role in advanced video-based surveillance systems, mainly because it is non intrusive and does not require people cooperation [1, 2]. Unfortunately, face recognition algorithms showed to suffer a lot from the high variability of environmental conditions. As an example, the effectiveness of face recognition strongly depends on lighting conditions and on variations in the subject’s pose and expression in front of the camera. This obviously limits their application to real video-surveillance tasks. On the other hand, face is considered a very good biometric. People recognize each other through the face, the acquisition process is non-intrusive, and does not require the collaboration of the subject to be recognized. Therefore, face recognition is a very active research field with many applications. For the purposes of this chapter, the face recognition applications can be subdivided in two types: applications in controlled and uncontrolled environments. One of the main applications of the first type is the so called “identity authentication”. A person submits to the automatic identity verification system its face (frontal and/or profile view) and declares her/his identity. The system matches the acquired face with the “template” stored in its data base, and classifies the person as a “genuine” (i.e., the claimed identity is accepted) or an “impostor”.
---
paper_title: Combining Pattern Classifiers: Methods and Algorithms

paper_content:

This book, which is wholly devoted to the subject of model combination, is divided into ten chapters. In addition to the first two introductory chapters, the book covers some of the following topics: multiple classifier systems; combination methods when the base classifier outputs are 0/1; methods when the outputs are continuous, e.g., posterior probabilities; methods for classifier selection; bagging and boosting; the theory of fixed combination rules; and the concept of diversity. Overall, it is a very well-written monograph. It explains and analyzes different approaches comparatively so that the reader can see how they are similar and how they differ. The literature survey is extensive. The MATLAB code for many methods is given in chapter appendices allowing readers to play with the explained methods or apply them quickly to their own data. The book is a must-read for researchers and practitioners alike.
---
paper_title: Taxonomy for characterizing ensemble methods in classification tasks: A review and annotated bibliography

paper_content:

Ensemble methodology, which builds a classification model by integrating multiple classifiers, can be used for improving prediction performance. Researchers from various disciplines such as statistics, pattern recognition, and machine learning have seriously explored the use of ensemble methodology. This paper presents an updated survey of ensemble methods in classification tasks, while introducing a new taxonomy for characterizing them. The new taxonomy, presented from the algorithm designer's point of view, is based on five dimensions: inducer, combiner, diversity, size, and members' dependency. We also propose several selection criteria, presented from the practitioner's point of view, for choosing the most suitable ensemble method.
---
paper_title: Methods of combining multiple classifiers and their applications to handwriting recognition

paper_content:

Possible solutions to the problem of combining classifiers can be divided into three categories according to the levels of information available from the various classifiers. Four approaches based on different methodologies are proposed for solving this problem. One is suitable for combining individual classifiers such as Bayesian, k-nearest-neighbor, and various distance classifiers. The other three could be used for combining any kind of individual classifiers. On applying these methods to combine several classifiers for recognizing totally unconstrained handwritten numerals, the experimental results show that the performance of individual classifiers can be improved significantly. For example, on the US zipcode database, 98.9% recognition with 0.90% substitution and 0.2% rejection can be obtained, as well as high reliability with 95% recognition, 0% substitution, and 5% rejection. >
---
paper_title: Optimal Linear Combinations Of Neural Networks

paper_content:

Neural network-based modeling often involves trying multiple networks with different architectures and training parameters in order to achieve acceptable model accuracy. Typically, one of the trained networks is chosen as best, while the rest are discarded. [Hashem and Schmeiser (1995)] proposed using optimal linear combinations of a number of trained neural networks instead of using a single best network. Combining the trained networks may help integrate the knowledge acquired by the components networks and thus improve model accuracy. In this paper, we extend the idea of optimal linear combinations (OLCs) of neural networks and discuss issues related to the generalization ability of the combined model. We then present two algorithms for selecting the component networks for the combination to improve the generalization ability of OLCs. Our experimental results demonstrate significant improvements in model accuracy, as a result of using OLCs, compared to using the apparent best network. Copyright 1997 Elsevier Science Ltd.
---
paper_title: Decision Combination in Multiple Classifier Systems

paper_content:

A multiple classifier system is a powerful solution to difficult pattern recognition problems involving large class sets and noisy input because it allows simultaneous use of arbitrary feature descriptors and classification procedures. Decisions by the classifiers can be represented as rankings of classifiers and different instances of a problem. The rankings can be combined by methods that either reduce or rerank a given set of classes. An intersection method and union method are proposed for class set reduction. Three methods based on the highest rank, the Borda count, and logistic regression are proposed for class set reranking. These methods have been tested in applications of degraded machine-printed characters and works from large lexicons, resulting in substantial improvement in overall correctness. >
---
paper_title: Statistical Pattern Recognition: A Review

paper_content:

The primary goal of pattern recognition is supervised or unsupervised classification. Among the various frameworks in which pattern recognition has been traditionally formulated, the statistical approach has been most intensively studied and used in practice. More recently, neural network techniques and methods imported from statistical learning theory have been receiving increasing attention. The design of a recognition system requires careful attention to the following issues: definition of pattern classes, sensing environment, pattern representation, feature extraction and selection, cluster analysis, classifier design and learning, selection of training and test samples, and performance evaluation. In spite of almost 50 years of research and development in this field, the general problem of recognizing complex patterns with arbitrary orientation, location, and scale remains unsolved. New and emerging applications, such as data mining, web searching, retrieval of multimedia data, face recognition, and cursive handwriting recognition, require robust and efficient pattern recognition techniques. The objective of this review paper is to summarize and compare some of the well-known methods used in various stages of a pattern recognition system and identify research topics and applications which are at the forefront of this exciting and challenging field.
---
paper_title: Combining forecasts: A review and annotated bibliography

paper_content:

Abstract Considerable literature has accumulated over the years regarding the combination of forecasts. The primary conclusion of this line of research is that forecast accuracy can be substantially improved through the combination of multiple individual forecasts. Furthermore, simple combination methods often work reasonably well relative to more complex combinations. This paper provides a review and annotated bibliography of that literature, including contributions from the forecasting, psychology, statistics, and management science literatures. The objectives are to provide a guide to the literature for students and researchers and to help researchers locate contributions in specific areas, both theoretical and applied. Suggestions for future research directions include (1) examination of simple combining approaches to determine reasons for their robustness, (2) development of alternative uses of multiple forecasts in order to make better use of the information they contain, (3) use of combined forecasts as benchmarks for forecast evaluation, and (4) study of subjective combination procedures. Finally, combining forecasts should become part of the mainstream of forecasting practice. In order to achieve this, practitioners should be encouraged to combine forecasts, and software to produce combined forecasts easily should be made available.
---
paper_title: A Two Stage Sequential Ensemble Applied to the Classification of Alzheimer’s Disease Based on MRI Features

paper_content:

We present a two stage sequential ensemble where data samples whose output from the first classifier fall in a low confidence output interval (LCOI) are processed by a second stage classifier. Training is composed of three processes: training the first classifier, determining the LCOI of the first classifier, and training the second classifier upon the data items whose output fall in the LCOI. The LCOI is determined varying a threshold on the false positive rate (FPR) and false negative rate (FNR) curves. We have tested the approach on a database of feature vectors for the classification of Alzheimer's disease (AD) and control subjects extracted from structural magnetic resonance imaging (sMRI) data. In this paper, we focus on the combinations obtained when the first classifier is a relevance vector machine (RVM). Obtained results improve over previous results for this database.
---
paper_title: Top 10 algorithms in data mining

paper_content:

This paper presents the top 10 data mining algorithms identified by the IEEE International Conference on Data Mining (ICDM) in December 2006: C4.5, k-Means, SVM, Apriori, EM, PageRank, AdaBoost, kNN, Naive Bayes, and CART. These top 10 algorithms are among the most influential data mining algorithms in the research community. With each algorithm, we provide a description of the algorithm, discuss the impact of the algorithm, and review current and further research on the algorithm. These 10 algorithms cover classification,
---
paper_title: Combining Diverse Neural Nets

paper_content:

An appropriate use of neural computing techniques is to apply them to problems such as condition monitoring, fault diagnosis, control and sensing, where conventional solutions can be hard to obtain. However, when neural computing techniques are used, it is important that they are employed so as to maximise their performance, and improve their reliability. Their performance is typically assessed in terms of their ability to generalise to a previously unseen test set, although unless the training set is very carefully chosen, 100p accuracy is rarely achieved. Improved performance can result when sets of neural nets are combined in ensembles and ensembles can be viewed as an example of the reliability through redundancy approach that is recommended for conventional software and hardware in safety-critical or safety-related applications. Although there has been recent interest in the use of neural net ensembles, such techniques have yet to be applied to the tasks of condition monitoring and fault diagnosis. In this paper, we focus on the benefits of techniques which promote diversity amongst the members of an ensemble, such that there is a minimum number of coincident failures. The concept of ensemble diversity is considered in some detail, and a hierarchy of four levels of diversity is presented. This hierarchy is then used in the description of the application of ensemble-based techniques to the case study of fault diagnosis of a diesel engine.
---
paper_title: good” and ”bad” diversity in majority vote ensembles

paper_content:

Although diversity in classifier ensembles is desirable, its relationship with the ensemble accuracy is not straightforward. Here we derive a decomposition of the majority vote error into three terms: average individual accuracy, “good” diversity and “bad diversity”. The good diversity term is taken out of the individual error whereas the bad diversity term is added to it. We relate the two diversity terms to the majority vote limits defined previously (the patterns of success and failure). A simulation study demonstrates how the proposed decomposition can be used to gain insights about majority vote classifier ensembles.
---
paper_title: Neural Network Ensembles, Cross Validation, and Active Learning

paper_content:

Learning of continuous valued functions using neural network ensembles (committees) can give improved accuracy, reliable estimation of the generalization error, and active learning. The ambiguity is defined as the variation of the output of ensemble members averaged over unlabeled data, so it quantifies the disagreement among the networks. It is discussed how to use the ambiguity in combination with cross-validation to give a reliable estimate of the ensemble generalization error, and how this type of ensemble cross-validation can sometimes improve performance. It is shown how to estimate the optimal weights of the ensemble members using unlabeled data. By a generalization of query by committee, it is finally shown how the ambiguity can be used to select new training data to be labeled in an active learning scheme.
---
paper_title: Selection of Heterogeneous Fuzzy Model Ensembles Using Self-adaptive Genetic Algorithms

paper_content:

The problem of model selection to compose a heterogeneous bagging ensemble was addressed in the paper. To solve the problem, three self-adapting genetic algorithms were proposed with different control parame- ters of mutation, crossover, and selection adjusted during the execution. The algorithms were applied to create heterogeneous ensembles comprising regres- sion fuzzy models to aid in real estate appraisals. The results of experiments revealed that the self-adaptive algorithms converged faster than the classic genetic algorithms. The heterogeneous ensembles created by self-adapting methods showed a very good predictive accuracy when compared with the homogeneous ensembles obtained in earlier research.
---
paper_title: Using Diversity in Preparing Ensembles of Classifiers Based on Different Feature Seature Subsets to Minimize Generalization Error

paper_content:

It is well known that ensembles of predictors produce better accuracy than a single predictor provided there is diversity in the ensemble. This diversity manifests itself as disagreement or ambiguity among the ensemble members. In this paper we focus on ensembles of classifiers based on different feature subsets and we present a process for producing such ensembles that emphasizes diversity (ambiguity) in the ensemble members. This emphasis on diversity produces ensembles with low generalization errors from ensemble members with comparatively high generalization error. We compare this with ensembles produced focusing only on the error of the ensemble members (without regard to overall diversity) and find that the ensembles based on ambiguity have lower generalization error. Further, we find that the ensemble members produced focusing on ambiguity have less features on average that those based on error only. We suggest that this indicates that these ensemble members are local learners.
---
paper_title: Software Diversity: Practical Statistics for its Measurement and Exploitation

paper_content:

Abstract The topic of this paper is the exploitation of diversity to enhance computer system reliability. It is well established that a diverse system composed of multiple alternative versions is more reliable than any single version alone, and this knowledge has occasionally been exploited in safety-critical applications. However, it is not clear what this property is, nor how the available diversity in a collection of versions is best exploited. We develop, define, illustrate and assess diversity measures, voting strategies for diversity exploitation, and interactions between the two. We take the view that a proper understanding of such issues is required if multiversion software engineering is to be elevated from the current “try it and see” procedure to a systematic technology. In addition, we introduce inductive programming techniques, particularly neural computing, as a cost-effective route to the practical use of multiversion systems outside the demanding requirements of safety-critical systems, i.e. in general software engineering.
---
paper_title: Diversity creation methods: a survey and categorisation

paper_content:

Ensemble approaches to classication and regression have attracted a great deal of interest in recent years. These methods can be shown both theoretically and empirically to outperform single predictors on a wide range of tasks. One of the elements required for accurate prediction when using an ensemble is recognised to be error \diversity". However, the exact meaning of this concept is not clear from the literature, particularly for classication tasks. In this paper we rst review the varied attempts to provide a formal explanation of error diversity, including several heuristic and qualitative explanations in the literature. For completeness of discussion we include not only the classication literature but also some excerpts of the rather more mature regression literature, which we believe can still provide some insights. We proceed to survey the various techniques used for creating diverse ensembles, and categorise them, forming a preliminary taxonomy of diversity creation methods. As part of this taxonomy we introduce the idea of implicit and explicit diversity creation methods, and three dimensions along which these may be applied. Finally we propose some new directions that may prove fruitful in understanding classication error diversity.
---
paper_title: Combining Diverse Neural Nets

paper_content:

An appropriate use of neural computing techniques is to apply them to problems such as condition monitoring, fault diagnosis, control and sensing, where conventional solutions can be hard to obtain. However, when neural computing techniques are used, it is important that they are employed so as to maximise their performance, and improve their reliability. Their performance is typically assessed in terms of their ability to generalise to a previously unseen test set, although unless the training set is very carefully chosen, 100p accuracy is rarely achieved. Improved performance can result when sets of neural nets are combined in ensembles and ensembles can be viewed as an example of the reliability through redundancy approach that is recommended for conventional software and hardware in safety-critical or safety-related applications. Although there has been recent interest in the use of neural net ensembles, such techniques have yet to be applied to the tasks of condition monitoring and fault diagnosis. In this paper, we focus on the benefits of techniques which promote diversity amongst the members of an ensemble, such that there is a minimum number of coincident failures. The concept of ensemble diversity is considered in some detail, and a hierarchy of four levels of diversity is presented. This hierarchy is then used in the description of the application of ensemble-based techniques to the case study of fault diagnosis of a diesel engine.
---
paper_title: Ensembling Neural Networks: Many Could Be Better Than All

paper_content:

Neural network ensemble is a learning paradigm where many neural networks are jointly used to solve a problem. In this paper, the relationship between the ensemble and its component neural networks is analyzed from the context of both regression and classification, which reveals that it may be better to ensemble many instead of all of the neural networks at hand. This result is interesting because at present, most approaches ensemble all the available neural networks for prediction. Then, in order to show that the appropriate neural networks for composing an ensemble can be effectively selected from a set of available neural networks, an approach named GASEN is presented. GASEN trains a number of neural networks at first. Then it assigns random weights to those networks and employs genetic algorithm to evolve the weights so that they can characterize to some extent the fitness of the neural networks in constituting an ensemble. Finally it selects some neural networks based on the evolved weights to make up the ensemble. A large empirical study shows that, compared with some popular ensemble approaches such as Bagging and Boosting, GASEN can generate neural network ensembles with far smaller sizes but stronger generalization ability. Furthermore, in order to understand the working mechanism of GASEN, the bias-variance decomposition of the error is provided in this paper, which shows that the success of GASEN may lie in that it can significantly reduce the bias as well as the variance.
---
paper_title: Relationships between combination methods and measures of diversity in combining classifiers

paper_content:

Abstract This study looks at the relationships between different methods of classifier combination and different measures of diversity. We considered 10 combination methods and 10 measures of diversity on two benchmark data sets. The relationship was sought on ensembles of three classifiers built on all possible partitions of the respective feature sets into subsets of pre-specified sizes. The only positive finding was that the Double-Fault measure of diversity and the measure of difficulty both showed reasonable correlation with Majority Vote and Naive Bayes combinations. Since both these measures have an indirect connection to the ensemble accuracy, this result was not unexpected. However, our experiments did not detect a consistent relationship between the other measures of diversity and the 10 combination methods.
---
paper_title: Cost-sensitive ensemble of support vector machines for effective detection of microcalcification in breast cancer diagnosis

paper_content:

This paper presents a new approach for the cost-sensitive classification problems based on the Boosting ensemble of support vector machines (SVMs). Different from conventional Boosting ensemble learning methods that adjust the distribution of training instances for minimizing the misclassification rate, the presented approach adjusts the training data distribution so as to minimize the expected cost of classification. This approach has been applied successfully in Microcalcification (MC) detection which is a typical cost-sensitive classification problem in breast cancer diagnosis. Its performance is evaluated by means of Receiver Operating Characteristics (ROC) curves and the expected costs of classification. Experimental results have consistently confirmed that the ROC of the SVM ensemble classifier is very close to the curve enveloping the base classifier ROC curves. This characteristic illustrates that the SVM ensemble is able to always improve the performance of the classification. Furthermore, the experimental results demonstrate that the method presented is able to not only increase the area under the ROC curve (AUC) but also minimize the expected classification cost.
---
paper_title: The Reliability of Dichotomous Judgments: Unequal Numbers of Judges per Subject

paper_content:

Consider a reliability study in which different subjects are judged on a dichotomous trait by dif ferent sets of judges, possibly unequal in number. A kappa-like measure of reliability is proposed, its correspondence to an intraclass correlation co efficient is pointed out, and a test for its statistical significance is presented. A numerical example is given.
---
paper_title: good” and ”bad” diversity in majority vote ensembles

paper_content:

Although diversity in classifier ensembles is desirable, its relationship with the ensemble accuracy is not straightforward. Here we derive a decomposition of the majority vote error into three terms: average individual accuracy, “good” diversity and “bad diversity”. The good diversity term is taken out of the individual error whereas the bad diversity term is added to it. We relate the two diversity terms to the majority vote limits defined previously (the patterns of success and failure). A simulation study demonstrates how the proposed decomposition can be used to gain insights about majority vote classifier ensembles.
---
paper_title: Limits on the majority vote accuracy in classifier fusion

paper_content:

We derive upper and lower limits on the majority vote accuracy with respect to individual accuracy p, the number of classiers in the pool (L), and the pairwise dependence between classiers, measured by Yule's Q statistic. Independence between individual classi- ers is typically viewed as an asset in classier fusion. We show that the majority vote with dependent classiers can potentially oer a dramatic improvement both over independent clas- siers and over the individual accuracy p. A functional relationship between the limits and the pairwise dependence Q is derived. Two patterns of the joint distribution for classier outputs (correct/incorrect) are identied to derive the limits: the pattern of success and the pattern of failure. The results support the intuition that negative pairwise dependence is benecial although not straightforwardly related to the accuracy. The pattern of success showed that for the highest improvement over p, all pairs of classiers in the pool should have the same negative dependence.
---
paper_title: Diversity versus Quality in Classification Ensembles based on Feature Selection

paper_content:

Feature subset-selection has emerged as a useful technique for creating diversity in ensembles - particularly in classification ensembles. In this paper we argue that this diversity needs to be monitored in the creation of the ensemble. We propose an entropy measure of the outputs of the ensemble members as a useful measure of the ensemble diversity. Further, we show that using the associated conditional entropy as a loss function (error measure) works well and the entropy in the ensemble predicts well the reduction in error due to the ensemble. These measures are evaluated on a medical prediction problem and are shown to predict the performance of the ensemble well. We also show that the entropy measure of diversity has the added advantage that it seems to model the change in diversity with the size of the ensemble.
---
paper_title: Pruning an ensemble of classifiers via reinforcement learning

paper_content:

This paper studies the problem of pruning an ensemble of classifiers from a reinforcement learning perspective. It contributes a new pruning approach that uses the Q-learning algorithm in order to approximate an optimal policy of choosing whether to include or exclude each classifier from the ensemble. Extensive experimental comparisons of the proposed approach against state-of-the-art pruning and combination methods show very promising results. Additionally, we present an extension that allows the improvement of the solutions returned by the proposed approach over time, which is very useful in certain performance-critical domains.
---
paper_title: Neural network ensembles

paper_content:

Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. >
---
paper_title: Managing Diversity in Regression Ensembles

paper_content:

Ensembles are a widely used and effective technique in machine learning---their success is commonly attributed to the degree of disagreement, or 'diversity', within the ensemble. For ensembles where the individual estimators output crisp class labels, this 'diversity' is not well understood and remains an open research issue. For ensembles of regression estimators, the diversity can be exactly formulated in terms of the covariance between individual estimator outputs, and the optimum level is expressed in terms of a bias-variance-covariance trade-off. Despite this, most approaches to learning ensembles use heuristics to encourage the right degree of diversity. In this work we show how to explicitly control diversity through the error function. The first contribution of this paper is to show that by taking the combination mechanism for the ensemble into account we can derive an error function for each individual that balances ensemble diversity with individual accuracy. We show the relationship between this error function and an existing algorithm called negative correlation learning, which uses a heuristic penalty term added to the mean squared error function. It is demonstrated that these methods control the bias-variance-covariance trade-off systematically, and can be utilised with any estimator capable of minimising a quadratic error function, for example MLPs, or RBF networks. As a second contribution, we derive a strict upper bound on the coefficient of the penalty term, which holds for any estimator that can be cast in a generalised linear regression framework, with mild assumptions on the basis functions. Finally we present the results of an empirical study, showing significant improvements over simple ensemble learning, and finding that this technique is competitive with a variety of methods, including boosting, bagging, mixtures of experts, and Gaussian processes, on a number of tasks.
---
paper_title: Classifier selection for majority voting

paper_content:

Abstract Individual classification models are recently challenged by combined pattern recognition systems, which often show better performance. In such systems the optimal set of classifiers is first selected and then combined by a specific fusion method. For a small number of classifiers optimal ensembles can be found exhaustively, but the burden of exponential complexity of such search limits its practical applicability for larger systems. As a result, simpler search algorithms and/or selection criteria are needed to reduce the complexity. This work provides a revision of the classifier selection methodology and evaluates the practical applicability of diversity measures in the context of combining classifiers by majority voting. A number of search algorithms are proposed and adjusted to work properly with a number of selection criteria including majority voting error and various diversity measures. Extensive experiments carried out with 15 classifiers on 27 datasets indicate inappropriateness of diversity measures used as selection criteria in favour of the direct combiner error based search. Furthermore, the results prompted a novel design of multiple classifier systems in which selection and fusion are recurrently applied to a population of best combinations of classifiers rather than the individual best. The improvement of the generalisation performance of such system is demonstrated experimentally.
---
paper_title: Neural Network Ensembles, Cross Validation, and Active Learning

paper_content:

Learning of continuous valued functions using neural network ensembles (committees) can give improved accuracy, reliable estimation of the generalization error, and active learning. The ambiguity is defined as the variation of the output of ensemble members averaged over unlabeled data, so it quantifies the disagreement among the networks. It is discussed how to use the ambiguity in combination with cross-validation to give a reliable estimate of the ensemble generalization error, and how this type of ensemble cross-validation can sometimes improve performance. It is shown how to estimate the optimal weights of the ensemble members using unlabeled data. By a generalization of query by committee, it is finally shown how the ambiguity can be used to select new training data to be labeled in an active learning scheme.
---
paper_title: Cost-Sensitive splitting and selection method for medical decision support system

paper_content:

The paper presents a cost-sensitive modification of the Adaptive Splitting and Selection (AdaSS) algorithm, which trains a combined classifier based on a feature space partitioning. In this study the algorithm considers constraints put on the cost of selected features, which are one of the key-problems in the clinical decision support systems. The modified version takes into consideration both the overall classification accuracy and the cost constraints, returning balanced solution for the problem at hand. Proposed method was evaluated on the basis of computer experiments run on cost-sensitive medical benchmark datasets.
---
paper_title: Design of Effective Neural Network Ensembles for Image Classification Purposes

paper_content:

Abstract In the field of pattern recognition, the combination of an ensemble of neural networks has been proposed as an approach to the development of high performance image classification systems. However, previous work clearly showed that such image classification systems are effective only if the neural networks forming them make different errors. Therefore, the fundamental need for methods aimed to design ensembles of ‘error-independent’ networks is currently acknowledged. In this paper, an approach to the automatic design of effective neural network ensembles is proposed. Given an initial large set of neural networks, our approach is aimed to select the subset formed by the most error-independent nets. Reported results on the classification of multisensor remote-sensing images show that this approach allows one to design effective neural network ensembles.
---
paper_title: Software Diversity: Practical Statistics for its Measurement and Exploitation

paper_content:

Abstract The topic of this paper is the exploitation of diversity to enhance computer system reliability. It is well established that a diverse system composed of multiple alternative versions is more reliable than any single version alone, and this knowledge has occasionally been exploited in safety-critical applications. However, it is not clear what this property is, nor how the available diversity in a collection of versions is best exploited. We develop, define, illustrate and assess diversity measures, voting strategies for diversity exploitation, and interactions between the two. We take the view that a proper understanding of such issues is required if multiversion software engineering is to be elevated from the current “try it and see” procedure to a systematic technology. In addition, we introduce inductive programming techniques, particularly neural computing, as a cost-effective route to the practical use of multiversion systems outside the demanding requirements of safety-critical systems, i.e. in general software engineering.
---
paper_title: Generalization error of ensemble estimators

paper_content:

It has been empirically shown that a better estimate with less generalization error can be obtained by averaging outputs of multiple estimators. This paper presents an analytical result for the generalization error of ensemble estimators. First, we derive a general expression of the ensemble generalization error by using factors of interest (bias, variance, covariance, and noise variance) and show how the generalization error is affected by each of them. Some special cases are then investigated. The result of a simulation is shown to verify our analytical result. A practically important problem of the ensemble approach, ensemble dilemma, is also discussed.
---
paper_title: Diversity creation methods: a survey and categorisation

paper_content:

Ensemble approaches to classication and regression have attracted a great deal of interest in recent years. These methods can be shown both theoretically and empirically to outperform single predictors on a wide range of tasks. One of the elements required for accurate prediction when using an ensemble is recognised to be error \diversity". However, the exact meaning of this concept is not clear from the literature, particularly for classication tasks. In this paper we rst review the varied attempts to provide a formal explanation of error diversity, including several heuristic and qualitative explanations in the literature. For completeness of discussion we include not only the classication literature but also some excerpts of the rather more mature regression literature, which we believe can still provide some insights. We proceed to survey the various techniques used for creating diverse ensembles, and categorise them, forming a preliminary taxonomy of diversity creation methods. As part of this taxonomy we introduce the idea of implicit and explicit diversity creation methods, and three dimensions along which these may be applied. Finally we propose some new directions that may prove fruitful in understanding classication error diversity.
---
paper_title: Ensemble diversity measures and their application to thinning

paper_content:

Abstract The diversity of an ensemble of classifiers can be calculated in a variety of ways. Here a diversity metric and a means for altering the diversity of an ensemble, called “thinning”, are introduced. We evaluate thinning algorithms created by several techniques on 22 publicly available datasets. When compared to other methods, our percentage correct diversity measure shows a greatest correlation between the increase in voted ensemble accuracy and the diversity value. Also, the analysis of different ensemble creation methods indicates that they generate different levels of diversity. Finally, the methods proposed for thinning show that ensembles can be made smaller without loss in accuracy.
---
paper_title: An analysis of diversity measures

paper_content:

Diversity among the base classifiers is deemed to be important when constructing a classifier ensemble. Numerous algorithms have been proposed to construct a good classifier ensemble by seeking both the accuracy of the base classifiers and the diversity among them. However, there is no generally accepted definition of diversity, and measuring the diversity explicitly is very difficult. Although researchers have designed several experimental studies to compare different diversity measures, usually confusing results were observed. In this paper, we present a theoretical analysis on six existing diversity measures (namely disagreement measure, double fault measure, KW variance, inter-rater agreement, generalized diversity and measure of difficulty), show underlying relationships between them, and relate them to the concept of margin, which is more explicitly related to the success of ensemble learning algorithms. We illustrate why confusing experimental results were observed and show that the discussed diversity measures are naturally ineffective. Our analysis provides a deeper understanding of the concept of diversity, and hence can help design better ensemble learning algorithms.
---
paper_title: Design of effective multiple classifier systems by clustering of classifiers

paper_content:

In the field of pattern recognition, multiple classifier systems based on the combination of outputs of a set of different classifiers have been proposed as a method for the development of high performance classification systems. Previous work clearly showed that multiple classifier. Systems are effective only if the classifiers forming them make independent errors. Therefore, the fundamental need for methods aimed to design "error-independent" classifiers is currently acknowledged. In the paper, an approach to the automatic design of multiple classifier systems is proposed. Given an initial large set of classifiers, our approach is aimed at selecting the subset formed by the most error-independent classifiers. Reported results on the classification of multisensor remote-sensing images show that this approach allows to design effective multiple classifier systems.
---
paper_title: Combining Pattern Classifiers: Methods and Algorithms

paper_content:

This book, which is wholly devoted to the subject of model combination, is divided into ten chapters. In addition to the first two introductory chapters, the book covers some of the following topics: multiple classifier systems; combination methods when the base classifier outputs are 0/1; methods when the outputs are continuous, e.g., posterior probabilities; methods for classifier selection; bagging and boosting; the theory of fixed combination rules; and the concept of diversity. Overall, it is a very well-written monograph. It explains and analyzes different approaches comparatively so that the reader can see how they are similar and how they differ. The literature survey is extensive. The MATLAB code for many methods is given in chapter appendices allowing readers to play with the explained methods or apply them quickly to their own data. The book is a must-read for researchers and practitioners alike.
---
paper_title: Letters: Experimental comparison of one-class classifiers for online signature verification

paper_content:

Online signature verification systems based on one-class classifiers are presented. Global information is extracted with a feature-based representation and recognized by using an ensemble of one-class classifiers. Experimental results obtained on the SUBCORPUS-100 MCYT signature database (100 signers, 5000 signatures) show that the machine experts, here proposed, outperform the state-of-the-art works both for random and skilled forgeries.
---
paper_title: Random decision forests

paper_content:

Decision trees are attractive classifiers due to their high execution speed. But trees derived with traditional methods often cannot be grown to arbitrary complexity for possible loss of generalization accuracy on unseen data. The limitation on complexity usually means suboptimal accuracy on training data. Following the principles of stochastic modeling, we propose a method to construct tree-based classifiers whose capacity can be arbitrarily expanded for increases in accuracy for both training and unseen data. The essence of the method is to build multiple trees in randomly selected subspaces of the feature space. Trees in, different subspaces generalize their classification in complementary ways, and their combined classification can be monotonically improved. The validity of the method is demonstrated through experiments on the recognition of handwritten digits.
---
paper_title: Fingerprint classification using one-vs-all support vector machines dynamically ordered with naïve Bayes classifiers

paper_content:

Fingerprint classification reduces the number of possible matches in automated fingerprint identification systems by categorizing fingerprints into predefined classes. Support vector machines (SVMs) are widely used in pattern classification and have produced high accuracy when performing fingerprint classification. In order to effectively apply SVMs to multi-class fingerprint classification systems, we propose a novel method in which the SVMs are generated with the one-vs-all (OVA) scheme and dynamically ordered with nai@?ve Bayes classifiers. This is necessary to break the ties that frequently occur when working with multi-class classification systems that use OVA SVMs. More specifically, it uses representative fingerprint features as the FingerCode, singularities and pseudo ridges to train the OVA SVMs and nai@?ve Bayes classifiers. The proposed method has been validated on the NIST-4 database and produced a classification accuracy of 90.8% for five-class classification with the statistical significance. The results show the benefits of integrating different fingerprint features as well as the usefulness of the proposed method in multi-class fingerprint classification.
---
paper_title: Building Decision Tree Classifier on Private Data

paper_content:

This paper studies how to build a decision tree classifier under the following scenario: a database is vertically partitioned into two pieces, with one piece owned by Alice and the other piece owned by Bob. Alice and Bob want to build a decision tree classifier based on such a database, but due to the privacy constraints, neither of them wants to disclose their private pieces to the other party or to any third party.We present a protocol that allows Alice and Bob to conduct such a classifier building without having to compromise their privacy. Our protocol uses an untrusted third-party server, and is built upon a useful building block, the scalar product protocol. Our solution to the scalar product protocol is more efficient than any existing solutions.
---
paper_title: Feature-subspace aggregating: ensembles for stable and unstable learners

paper_content:

This paper introduces a new ensemble approach, Feature-Subspace Aggregating (Feating), which builds local models instead of global models. Feating is a generic ensemble approach that can enhance the predictive performance of both stable and unstable learners. In contrast, most existing ensemble approaches can improve the predictive performance of unstable learners only. Our analysis shows that the new approach reduces the execution time to generate a model in an ensemble through an increased level of localisation in Feating. Our empirical evaluation shows that Feating performs significantly better than Boosting, Random Subspace and Bagging in terms of predictive accuracy, when a stable learner SVM is used as the base learner. The speed up achieved by Feating makes feasible SVM ensembles that would otherwise be infeasible for large data sets. When SVM is the preferred base learner, we show that Feating SVM performs better than Boosting decision trees and Random Forests. We further demonstrate that Feating also substantially reduces the error of another stable learner, k-nearest neighbour, and an unstable learner, decision tree.
---
paper_title: The random subspace method for constructing decision forests

paper_content:

Much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes. The dilemma between overfitting and achieving maximum accuracy is seldom resolved. A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity. The classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector, that is, trees constructed in randomly chosen subspaces. The subspace method is compared to single-tree classifiers and other forest construction methods by experiments on publicly available datasets, where the method's superiority is demonstrated. We also discuss independence between trees in a forest and relate that to the combined classification accuracy.
---
paper_title: A study on the performances of dynamic classifier selection based on local accuracy estimation

paper_content:

Dynamic classifier selection (DCS) plays a strategic role in the field of multiple classifier systems (MCS). This paper proposes a study on the performances of DCS by Local Accuracy estimation (DCS-LA). To this end, upper bounds against which the performances can be evaluated are proposed. The experimental results on five datasets clearly show the effectiveness of the selection methods based on local accuracy estimates.
---
paper_title: Effective classification of noisy data streams with attribute-oriented dynamic classifier selection

paper_content:

Recently, mining from data streams has become an important and challenging task for many real-world applications such as credit card fraud protection and sensor networking. One popular solution is to separate stream data into chunks, learn a base classifier from each chunk, and then integrate all base classifiers for effective classification. In this paper, we propose a new dynamic classifier selection (DCS) mechanism to integrate base classifiers for effective mining from data streams. The proposed algorithm dynamically selects a single “best” classifier to classify each test instance at run time. Our scheme uses statistical information from attribute values, and uses each attribute to partition the evaluation set into disjoint subsets, followed by a procedure that evaluates the classification accuracy of each base classifier on these subsets. Given a test instance, its attribute values determine the subsets that the similar instances in the evaluation set have constructed, and the classifier with the highest classification accuracy on those subsets is selected to classify the test instance. Experimental results and comparative studies demonstrate the efficiency and efficacy of our method. Such a DCS scheme appears to be promising in mining data streams with dramatic concept drifting or with a significant amount of noise, where the base classifiers are likely conflictive or have low confidence.
---
paper_title: Decentralized distributed computing system for privacy-preserving combined classifiers: modeling and optimization

paper_content:

The growing amount of various kinds of information triggers the need to develop efficient network computing systems, as single machines in many cases are not able to provide effective processing and analysis. One of the very promising approaches of distributed data analysis is combined classification, which could be relatively easily implemented in distributed computing systems. In this paper we address problem of decentralized distributed computing system for mentioned above classification method. We focus on the system fairness. The performance metric is defined as a maximum response time, i.e., the computing system should be designed to minimize the response time of each client using the system. We assume that the system is decentralized and each request is sent by the client directly to computing nodes without assistance of a central service. An ILP (Integer Linear Programming) model is formulated and applied to obtain optimal results provided by branch-and-cut algorithm included in the CPLEX solver. Widespread simulations are performed to evaluate properties of the computing system in terms of several parameters describing the system.
---
paper_title: Privacy Preserving Models of k -NN Algorithm

paper_content:

The paper focuses on the problem of privacy preserving for classification task. This issue is quite an important subject for the machine learning approach based on distributed databases. On the basis of the study of available works devoted to privacy we propose its new definition and its taxonomy. We use this taxonomy to create several modifications of k-nearest neighbors classifier which are consistent with the proposed privacy levels. Their computational complexity are evaluated on the basis of computer experiments.
---
paper_title: From dynamic classifier selection to dynamic ensemble selection

paper_content:

In handwritten pattern recognition, the multiple classifier system has been shown to be useful for improving recognition rates. One of the most important tasks in optimizing a multiple classifier system is to select a group of adequate classifiers, known as an Ensemble of Classifiers (EoC), from a pool of classifiers. Static selection schemes select an EoC for all test patterns, and dynamic selection schemes select different classifiers for different test patterns. Nevertheless, it has been shown that traditional dynamic selection performs no better than static selection. We propose four new dynamic selection schemes which explore the properties of the oracle concept. Our results suggest that the proposed schemes, using the majority voting rule for combining classifiers, perform better than the static selection method.
---
paper_title: Empirical comparison of Dynamic Classifier Selection methods based on diversity and accuracy for building ensembles

paper_content:

In the context of Ensembles or Multi-Classifier Systems, the choice of the ensemble members is a very complex task, in which, in some cases, it can lead to ensembles with no performance improvement. In order to avoid this situation, there is a great deal of research to find effective classifier member selection methods. In this paper, we propose a selection criterion based on both the accuracy and diversity of the classifiers in the initial pool. Also, instead of using a static selection method, we use a Dynamic Classifier Selection (DSC) procedure. In this case, the member classifiers to form the ensemble are chosen at the test (use) phase. That is, different testing patterns can be classified by different ensemble configurations.
---
paper_title: A comparison of approaches to large-scale data analysis

paper_content:

There is currently considerable enthusiasm around the MapReduce (MR) paradigm for large-scale data analysis [17]. Although the basic control flow of this framework has existed in parallel SQL database management systems (DBMS) for over 20 years, some have called MR a dramatically new computing model [8, 17]. In this paper, we describe and compare both paradigms. Furthermore, we evaluate both kinds of systems in terms of performance and development complexity. To this end, we define a benchmark consisting of a collection of tasks that we have run on an open source version of MR as well as on two parallel DBMSs. For each task, we measure each system's performance for various degrees of parallelism on a cluster of 100 nodes. Our results reveal some interesting trade-offs. Although the process to load data into and tune the execution of parallel DBMSs took much longer than the MR system, the observed performance of these DBMSs was strikingly better. We speculate about the causes of the dramatic performance difference and consider implementation concepts that future systems should take from both kinds of architectures.
---
paper_title: Nearest neighbor classification from multiple feature subsets

paper_content:

Combining multiple classifiers is an effective technique for improving accuracy. There are many general combining algorithms, such as Bagging, Boosting, or Error Correcting Output Coding, that significantly improve classifiers like decision trees, rule learners, or neural networks. Unfortunately, these combining methods do not improve the nearest neighbor classifier. In this paper, we present MFS, a combining algorithm designed to improve the accuracy of the nearest neighbor NN classifier. MFS combines multiple NN classifiers each using only a random subset of features. The experimental results are encouraging: On 25 datasets from the UCI repository, MFS significantly outperformed several standard NN variants and was competitive with boosted decision trees. In additional experiments, we show that MFS is robust to irrelevant features, and is able to reduce both bias and variance components of error.
---
paper_title: Optimizing nearest neighbour in random subspaces using a multi-objective genetic algorithm

paper_content:

In this work, the authors have evaluated almost 20 millions ensembles of classifiers generated by several methods. Trying to optimize those ensembles based on the nearest neighbours and the random subspaces paradigms, we found that the use of a diversity metric called "ambiguity" had no better positive impact than plain stochastic search.
---
paper_title: Clustering-and-selection model for classifier combination

paper_content:

We devise a simple clustering-and-selection algorithm based on a probabilistic interpretation of classifier selection. First, the data set is clustered into K clusters, and then the most successful classifier for a given cluster is nominated to label the inputs in the Voronoi cell of the cluster centroid. The proposed method is compared experimentally with the minimum, maximum, product and average. Also given are the results from the naive Bayes method, the behaviour-knowledge space (BKS) method, the best individual and the oracle.
---
paper_title: Classifier selection for majority voting

paper_content:

Abstract Individual classification models are recently challenged by combined pattern recognition systems, which often show better performance. In such systems the optimal set of classifiers is first selected and then combined by a specific fusion method. For a small number of classifiers optimal ensembles can be found exhaustively, but the burden of exponential complexity of such search limits its practical applicability for larger systems. As a result, simpler search algorithms and/or selection criteria are needed to reduce the complexity. This work provides a revision of the classifier selection methodology and evaluates the practical applicability of diversity measures in the context of combining classifiers by majority voting. A number of search algorithms are proposed and adjusted to work properly with a number of selection criteria including majority voting error and various diversity measures. Extensive experiments carried out with 15 classifiers on 27 datasets indicate inappropriateness of diversity measures used as selection criteria in favour of the direct combiner error based search. Furthermore, the results prompted a novel design of multiple classifier systems in which selection and fusion are recurrently applied to a population of best combinations of classifiers rather than the individual best. The improvement of the generalisation performance of such system is demonstrated experimentally.
---
paper_title: Attribute bagging : improving accuracy of classi # er ensembles by using random feature subsets

paper_content:

We present attribute bagging (AB), a technique for improving the accuracy and stability of classifier ensembles induced using random subsets of features. AB is a wrapper method that can be used with any learning algorithm. It establishes an appropriate attribute subset size and then randomly selects subsets of features, creating projections of the training set on which the ensemble classifiers are built. The induced classifiers are then used for voting. This article compares the performance of our AB method with bagging and other algorithms on a hand-pose recognition dataset. It is shown that AB gives consistently better results than bagging, both in accuracy and stability. The performance of ensemble voting in bagging and the AB method as a function of the attribute subset size and the number of voters for both weighted and unweighted voting is tested and discussed. We also demonstrate that ranking the attribute subsets by their classification accuracy and voting using only the best subsets further improves the resulting performance of the ensemble.
---
paper_title: Dynamic integration of classifiers for handling concept drift

paper_content:

In the real world concepts are often not stable but change with time. A typical example of this in the biomedical context is antibiotic resistance, where pathogen sensitivity may change over time as new pathogen strains develop resistance to antibiotics that were previously effective. This problem, known as concept drift, complicates the task of learning a model from data and requires special approaches, different from commonly used techniques that treat arriving instances as equally important contributors to the final concept. The underlying data distribution may change as well, making previously built models useless. This is known as virtual concept drift. Both types of concept drifts make regular updates of the model necessary. Among the most popular and effective approaches to handle concept drift is ensemble learning, where a set of models built over different time periods is maintained and the best model is selected or the predictions of models are combined, usually according to their expertise level regarding the current concept. In this paper we propose the use of an ensemble integration technique that would help to better handle concept drift at an instance level. In dynamic integration of classifiers, each base classifier is given a weight proportional to its local accuracy with regard to the instance tested, and the best base classifier is selected, or the classifiers are integrated using weighted voting. Our experiments with synthetic data sets simulating abrupt and gradual concept drifts and with a real-world antibiotic resistance data set demonstrate that dynamic integration of classifiers built over small time intervals or fixed-sized data blocks can be significantly better than majority voting and weighted voting, which are currently the most commonly used integration techniques for handling concept drift with ensembles.
---
paper_title: Random Forests

paper_content:

Random forests are a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest. The generalization error for forests converges a.s. to a limit as the number of trees in the forest becomes large. The generalization error of a forest of tree classifiers depends on the strength of the individual trees in the forest and the correlation between them. Using a random selection of features to split each node yields error rates that compare favorably to Adaboost (Y. Freund & R. Schapire, Machine Learning: Proceedings of the Thirteenth International conference, aaa, 148–156), but are more robust with respect to noise. Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the splitting. Internal estimates are also used to measure variable importance. These ideas are also applicable to regression.
---
paper_title: An automatic construction and organization strategy for ensemble learning on data streams

paper_content:

As data streams are gaining prominence in a growing number of emerging application domains, classification on data streams is becoming an active research area. Currently, the typical approach to this problem is based on ensemble learning, which learns basic classifiers from training data stream and forms the global predictor by organizing these basic ones. While this approach seems successful to some extent, its performance usually suffers from two contradictory elements existing naturally within many application scenarios: firstly, the need for gathering sufficient training data for basic classifiers and engaging enough basic learners in voting for bias-variance reduction; and secondly, the requirement for significant sensitivity to concept-drifts, which places emphasis on using recent training data and up-to-date individual classifiers. It results in such a dilemma that some algorithms are not sensitive enough to concept-drifts while others, although sensitive enough, suffer from unsatisfactory classification accuracy. In this paper, we propose an ensemble learning algorithm, which: (1) furnishes training data for basic classifiers, starting from the up-to-date data chunk and searching for complement from past chunks while ruling out the data inconsistent with current concept; (2) provides effective voting by adaptively distinguishing sensible classifiers from the else and engaging sensible ones as voters. Experimental results justify the superiority of this strategy in terms of both accuracy and sensitivity, especially in severe circumstances where training data is extremely insufficient or concepts are evolving frequently and significantly.
---
paper_title: Neural Network Ensembles, Cross Validation, and Active Learning

paper_content:

Learning of continuous valued functions using neural network ensembles (committees) can give improved accuracy, reliable estimation of the generalization error, and active learning. The ambiguity is defined as the variation of the output of ensemble members averaged over unlabeled data, so it quantifies the disagreement among the networks. It is discussed how to use the ambiguity in combination with cross-validation to give a reliable estimate of the ensemble generalization error, and how this type of ensemble cross-validation can sometimes improve performance. It is shown how to estimate the optimal weights of the ensemble members using unlabeled data. By a generalization of query by committee, it is finally shown how the ambiguity can be used to select new training data to be labeled in an active learning scheme.
---
paper_title: Bagging Predictors

paper_content:

Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.
---
paper_title: Algorithm of designing compound recognition system on the basis of combining classifiers with simultaneous splitting feature space into competence areas

paper_content:

The paper presents the novel adaptive splitting and selection algorithm (AdaSS) used for learning compound pattern recognition system. Splitting a feature space into its constituents and selection of the best area classifier from the pool of available recognizers for each region are key processes of the proposed model. Both take place simultaneously as part of a compound optimization process aimed at maximizing system performance. Evolutionary algorithms are used to find out the optimal solution. The results of experiments for algorithm evaluation purposes prove the quality of the proposed approach.
---
paper_title: A Cascaded Multiple Expert System for Verification

paper_content:

In this paper we propose a multi-expert architecture, particularly suited for verification systems, which attempts to offer the performance advantages of a serial approach while retaining the reliability of a parallel combination scheme. In this framework, criteria for evaluating the reliability of both the intermediate answers and the final response are provided, together with a method for the determination of an optimal reject option. This architecture has been tested on a signature verification application, confirming the effectiveness of the proposed approach.
---
paper_title: Privacy-preserving data mining

paper_content:

A fruitful direction for future data mining research will be the development of techniques that incorporate privacy concerns. Specifically, we address the following question. Since the primary task in data mining is the development of models about aggregated data, can we develop accurate models without access to precise information in individual data records? We consider the concrete case of building a decision-tree classifier from training data in which the values of individual records have been perturbed. The resulting data records look very different from the original records and the distribution of data values is also very different from the original distribution. While it is not possible to accurately estimate original values in individual data records, we propose a novel reconstruction procedure to accurately estimate the distribution of original data values. By using these reconstructed distributions, we are able to build classifiers whose accuracy is comparable to the accuracy of classifiers built with the original data.
---
paper_title: Design of effective multiple classifier systems by clustering of classifiers

paper_content:

In the field of pattern recognition, multiple classifier systems based on the combination of outputs of a set of different classifiers have been proposed as a method for the development of high performance classification systems. Previous work clearly showed that multiple classifier. Systems are effective only if the classifiers forming them make independent errors. Therefore, the fundamental need for methods aimed to design "error-independent" classifiers is currently acknowledged. In the paper, an approach to the automatic design of multiple classifier systems is proposed. Given an initial large set of classifiers, our approach is aimed at selecting the subset formed by the most error-independent classifiers. Reported results on the classification of multisensor remote-sensing images show that this approach allows to design effective multiple classifier systems.
---
paper_title: Mining concept-drifting data streams using ensemble classifiers

paper_content:

Recently, mining data streams with concept drifts for actionable insights has become an important and challenging task for a wide range of applications including credit card fraud protection, target marketing, network intrusion detection, etc. Conventional knowledge discovery tools are facing two challenges, the overwhelming volume of the streaming data, and the concept drifts. In this paper, we propose a general framework for mining concept-drifting data streams using weighted ensemble classifiers. We train an ensemble of classification models, such as C4.5, RIPPER, naive Beyesian, etc., from sequential chunks of the data stream. The classifiers in the ensemble are judiciously weighted based on their expected classification accuracy on the test data under the time-evolving environment. Thus, the ensemble approach improves both the efficiency in learning the model and the accuracy in performing classification. Our empirical study shows that the proposed methods have substantial advantage over single-classifier approaches in prediction accuracy, and the ensemble framework is effective for a variety of classification models.
---
paper_title: Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval

paper_content:

Relevance feedback schemes based on support vector machines (SVM) have been widely used in content-based image retrieval (CBIR). However, the performance of SVM-based relevance feedback is often poor when the number of labeled positive feedback samples is small. This is mainly due to three reasons: 1) an SVM classifier is unstable on a small-sized training set, 2) SVM's optimal hyperplane may be biased when the positive feedback samples are much less than the negative feedback samples, and 3) overfitting happens because the number of feature dimensions is much higher than the size of the training set. In this paper, we develop a mechanism to overcome these problems. To address the first two problems, we propose an asymmetric bagging-based SVM (AB-SVM). For the third problem, we combine the random subspace method and SVM for relevance feedback, which is named random subspace SVM (RS-SVM). Finally, by integrating AB-SVM and RS-SVM, an asymmetric bagging and random subspace SVM (ABRS-SVM) is built to solve these three problems and further improve the relevance feedback performance
---
paper_title: A probabilistic model of classifier competence for dynamic ensemble selection

paper_content:

The concept of a classifier competence is fundamental to multiple classifier systems (MCSs). In this study, a method for calculating the classifier competence is developed using a probabilistic model. In the method, first a randomised reference classifier (RRC) whose class supports are realisations of the random variables with beta probability distributions is constructed. The parameters of the distributions are chosen in such a way that, for each feature vector in a validation set, the expected values of the class supports produced by the RRC and the class supports produced by a modelled classifier are equal. This allows for using the probability of correct classification of the RRC as the competence of the modelled classifier. The competences calculated for a validation set are then generalised to an entire feature space by constructing a competence function based on a potential function model or regression. Three systems based on a dynamic classifier selection and a dynamic ensemble selection (DES) were constructed using the method developed. The DES based system had statistically significant higher average rank than the ones of eight benchmark MCSs for 22 data sets and a heterogeneous ensemble. The results obtained indicate that the full vector of class supports should be used for evaluating the classifier competence as this potentially improves performance of MCSs.
---
paper_title: A streaming ensemble algorithm (SEA) for large-scale classification

paper_content:

Ensemble methods have recently garnered a great deal of attention in the machine learning community. Techniques such as Boosting and Bagging have proven to be highly effective but require repeated resampling of the training data, making them inappropriate in a data mining context. The methods presented in this paper take advantage of plentiful data, building separate classifiers on sequential chunks of training points. These classifiers are combined into a fixed-size ensemble using a heuristic replacement strategy. The result is a fast algorithm for large-scale or streaming data that classifies as well as a single decision tree built on all the data, requires approximately constant memory, and adjusts quickly to concept drift.
---
paper_title: An efficient fuzzy classifier with feature selection based on fuzzy entropy

paper_content:

This paper presents an efficient fuzzy classifier with the ability of feature selection based on a fuzzy entropy measure. Fuzzy entropy is employed to evaluate the information of pattern distribution in the pattern space. With this information, we can partition the pattern space into nonoverlapping decision regions for pattern classification. Since the decision regions do not overlap, both the complexity and computational load of the classifier are reduced and thus the training time and classification time are extremely short. Although the decision regions are partitioned into nonoverlapping subspaces, we can achieve good classification performance since the decision regions can be correctly determined via our proposed fuzzy entropy measure. In addition, we also investigate the use of fuzzy entropy to select relevant features. The feature selection procedure not only reduces the dimensionality of a problem but also discards noise-corrupted, redundant and unimportant features. Finally, we apply the proposed classifier to the Iris database and Wisconsin breast cancer database to evaluate the classification performance. Both of the results show that the proposed classifier can work well for the pattern classification application.
---
paper_title: Dynamic weighted majority: a new ensemble method for tracking concept drift

paper_content:

Algorithms for tracking concept drift are important for many applications. We present a general method based on the weighted majority algorithm for using any online learner for concept drift. Dynamic weighted majority (DWM) maintains an ensemble of base learners, predicts using a weighted-majority vote of these "experts", and dynamically creates and deletes experts in response to changes in performance. We empirically evaluated two experimental systems based on the method using incremental naive Bayes and incremental tree inducer [ITI] as experts. For the sake of comparison, we also included Blum's implementation of weighted majority. On the STAGGER concepts and on the SEA concepts, results suggest that the ensemble method learns drifting concepts almost as well as the base algorithms learn each concept individually. Indeed, we report the best overall results for these problems to date.
---
paper_title: Hybrid Classification Ensemble Using Topology-preserving Clustering

paper_content:

This study presents a novel hybrid intelligent system using both unsupervised and supervised learning that can be easily adapted to be used in an individual or collaborative system. The system divides the classification problem into two stages: firstly it divides the input data space into different parts, according to the input space distribution of the data set. Then, it generates several simple classifiers that are used to correctly classify samples that are contained in one of the previously determined parts. This way, the efficiency of each classifier increases, as they can specialize in classifying only related samples from certain regions of the input data space. This specialization of the single classifiers enables them to learn more specific patterns or characteristics of the data space, avoiding the risk of obtaining a general algorithm that over-fits to the data. The hybrid system presented has been tested with artificial and real data sets. A comparative study of the results obtained by the novel model with those obtained from other common classification methods is also included in the present work.
---
paper_title: Partial classification: the benefit of deferred decision

paper_content:

It is shown that partial classification which allows for indecision in certain regions of the data space, can increase a benefit function, defined as the difference between the probabilities of correct and incorrect decisions, joint with the event that a decision is made. This is particularly true for small data samples, which may cause a large deviation of the estimated separation surface from the intersection surface between the corresponding probability density functions. Employing a particular density estimation method, an indecision domain is naturally defined by a single parameter whose optimal size, maximizing the benefit function, is derived from the data. The benefit function is shown to translate into profit in stock trading. Employing medical and economic data, it is shown that partial classification produces, on average, higher benefit values than full classification, assigning each new object to a class, and that the marginal benefit of partial classification reduces as the data size increases.
---
paper_title: Combined classifier based on feature space partitioning

paper_content:

This paper presents a significant modification to the AdaSS Adaptive Splitting and Selection algorithm, which was developed several years ago. The method is based on the simultaneous partitioning of the feature space and an assignment of a compound classifier to each of the subsets. The original version of the algorithm uses a classifier committee and a majority voting rule to arrive at a decision. The proposed modification replaces the fairly simple fusion method with a combined classifier, which makes a decision based on a weighted combination of the discriminant functions of the individual classifiers selected for the committee. The weights mentioned above are dependent not only on the classifier identifier, but also on the class number. The proposed approach is based on the results of previous works, where it was proven that such a combined classifier method could achieve significantly better results than simple voting systems. The proposed modification was evaluated through computer experiments, carried out on diverse benchmark datasets. The results are very promising in that they show that, for most of the datasets, the proposed method outperforms similar techniques based on the clustering and selection approach.
---
paper_title: Bagging, Boosting and the Random Subspace Method for Linear Classifiers

paper_content:

Recently bagging, boosting and the random subspace method have become popular combining techniques for improving weak classifiers. These techniques are designed for, and usually applied to, decision trees. In this paper, in contrast to a common opinion, we demonstrate that they may also be useful in linear discriminant analysis. Simulation studies, carried out for several artificial and real data sets, show that the performance of the combining techniques is strongly affected by the small sample size properties of the base classifier: boosting is useful for large training sample sizes, while bagging and the random subspace method are useful for critical training sample sizes. Finally, a table describing the possible usefulness of the combining techniques for linear classifiers is presented.
---
paper_title: A measure of competence based on random classification for dynamic ensemble selection

paper_content:

In this paper, a measure of competence based on random classification (MCR) for classifier ensembles is presented. The measure selects dynamically (i.e. for each test example) a subset of classifiers from the ensemble that perform better than a random classifier. Therefore, weak (incompetent) classifiers that would adversely affect the performance of a classification system are eliminated. When all classifiers in the ensemble are evaluated as incompetent, the classification accuracy of the system can be increased by using the random classifier instead. Theoretical justification for using the measure with the majority voting rule is given. Two MCR based systems were developed and their performance was compared against six multiple classifier systems using data sets taken from the UCI Machine Learning Repository and Ludmila Kuncheva Collection. The systems developed had typically the highest classification accuracies regardless of the ensemble type used (homogeneous or heterogeneous).
---
paper_title: The Boosting Approach to Machine Learning An Overview

paper_content:

Boosting is a general method for improving the accuracy of any given learning algorithm. Focusing primarily on the AdaBoost algorithm, this chapter overviews some of the recent work on boosting including analyses of AdaBoost’s training error and generalization error; boosting’s connection to game theory and linear programming; the relationship between boosting and logistic regression; extensions of AdaBoost for multiclass classification problems; methods of incorporating human knowledge into boosting; and experimental and applied work using boosting.
---
paper_title: Probability Estimates for Multi-class Classification by Pairwise Coupling

paper_content:

Pairwise coupling is a popular multi-class classification method that combines together all pairwise comparisons for each pair of classes. This paper presents two approaches for obtaining class probabilities. Both methods can be reduced to linear systems and are easy to implement. We show conceptually and experimentally that the proposed approaches are more stable than two existing popular methods: voting and [3].
---
paper_title: Combining One-Class Classifiers

paper_content:

In the problem of one-class classification target objects should be distinguished from outlier objects. In this problem it is assumed that only information of the target class is available while nothing is known about the outlier class. Like standard two-class classifiers, one-class classifiers hardly ever fit the data distribution perfectly. Using only the best classifier and discarding the classifiers with poorer performance might waste valuable information. To improve performance the results of different classifiers (which may differ in complexity or training algorithm) can be combined. This can not only increase the performance but it can also increase the robustness of the classification. Because for one-class classifiers only information of one of the classes is present, combining one-class classifiers is more difficult. In this paper we investigate if and how one-class classifiers can be combined best in a handwritten digit recognition problem.
---
paper_title: New results on error correcting output codes of kernel machines

paper_content:

We study the problem of multiclass classification within the framework of error correcting output codes (ECOC) using margin-based binary classifiers. Specifically, we address two important open problems in this context: decoding and model selection. The decoding problem concerns how to map the outputs of the classifiers into class codewords. In this paper we introduce a new decoding function that combines the margins through an estimate of their class conditional probabilities. Concerning model selection, we present new theoretical results bounding the leave-one-out (LOO) error of ECOC of kernel machines, which can be used to tune kernel hyperparameters. We report experiments using support vector machines as the base binary classifiers, showing the advantage of the proposed decoding function over other functions of I he margin commonly used in practice. Moreover, our empirical evaluations on model selection indicate that the bound leads to good estimates of kernel parameters.
---
paper_title: Intrusion detection in computer networks by a modular ensemble of one-class classifiers

paper_content:

Since the early days of research on intrusion detection, anomaly-based approaches have been proposed to detect intrusion attempts. Attacks are detected as anomalies when compared to a model of normal (legitimate) events. Anomaly-based approaches typically produce a relatively large number of false alarms compared to signature-based IDS. However, anomaly-based IDS are able to detect never-before-seen attacks. As new types of attacks are generated at an increasing pace and the process of signature generation is slow, it turns out that signature-based IDS can be easily evaded by new attacks. The ability of anomaly-based IDS to detect attacks never observed in the wild has stirred up a renewed interest in anomaly detection. In particular, recent work focused on unsupervised or unlabeled anomaly detection, due to the fact that it is very hard and expensive to obtain a labeled dataset containing only pure normal events. The unlabeled approaches proposed so far for network IDS focused on modeling the normal network traffic considered as a whole. As network traffic related to different protocols or services exhibits different characteristics, this paper proposes an unlabeled Network Anomaly IDS based on a modular Multiple Classifier System (MCS). Each module is designed to model a particular group of similar protocols or network services. The use of a modular MCS allows the designer to choose a different model and decision threshold for different (groups of) network services. This also allows the designer to tune the false alarm rate and detection rate produced by each module to optimize the overall performance of the ensemble. Experimental results on the KDD-Cup 1999 dataset show that the proposed anomaly IDS achieves high attack detection rate and low false alarm rate at the same time.
---
paper_title: Multi-category Classification by Soft-Max Combination of Binary Classifiers

paper_content:

In this paper, we propose a multi-category classification method that combines binary classifiers through soft-max function. Posteriori probabilities are also obtained. Both, one-versus-all and one-versus-one classifiers can be used in the combination. Empirical comparison shows that the proposed method is competitive with other implementations of one-versus-all and one-versus-one methods in terms of both classification accuracy and posteriori probability estimate.
---
paper_title: Using two-class classifiers for multiclass classification

paper_content:

The generalization from two-class classification to multiclass classification is not straightforward for discriminants which are not based on density estimation. Simple combining methods use voting, but this has the drawback of inconsequent labelings and ties. More advanced methods map the discriminant outputs to approximate posterior probability estimates and combine these, while other methods use error-correcting output codes. In this paper we want to show the possibilities of simple generalizations of the two-class classification, using voting and combinations of approximate posterior probabilities.
---
paper_title: Combining Predictions in Pairwise Classification: An Optimal Adaptive Voting Strategy and Its Relation to Weighted Voting

paper_content:

Weighted voting is the commonly used strategy for combining predictions in pairwise classification. Even though it shows good classification performance in practice, it is often criticized for lacking a sound theoretical justification. In this paper, we study the problem of combining predictions within a formal framework of label ranking and, under some model assumptions, derive a generalized voting strategy in which predictions are properly adapted according to the strengths of the corresponding base classifiers. We call this strategy adaptive voting and show that it is optimal in the sense of yielding a MAP prediction of the class label of a test instance. Moreover, we offer a theoretical justification for weighted voting by showing that it yields a good approximation of the optimal adaptive voting prediction. This result is further corroborated by empirical evidence from experiments with real and synthetic data sets showing that, even though adaptive voting is sometimes able to achieve consistent improvements, weighted voting is in general quite competitive, all the more in cases where the aforementioned model assumptions underlying adaptive voting are not met. In this sense, weighted voting appears to be a more robust aggregation strategy.
---
paper_title: Soft computing methods applied to combination of one-class classifiers

paper_content:

The paper shows the possibilities of generalizing the two-class classification into multi-class classification by means of a fuzzy inference system. Fuzzy combiner harnesses the support values from classifiers to provide final response having no other restrictions on their structure. We compare proposed combination methods with ECOC and two variations of decision templates, based on Euclidean and symmetric distance. The effectiveness of the proposed combination method based on the fuzzy logic theory is also evaluated via computer experiments carried out on benchmark datasets.
---
paper_title: Solving Multiclass Learning Problems via Error-Correcting Output Codes

paper_content:

Multiclass learning problems involve finding a definition for an unknown function f(x) whose range is a discrete set containing k > 2 values (i.e., k "classes"). The definition is acquired by studying collections of training examples of the form (xi, f(xi)). Existing approaches to multiclass learning problems include direct application of multiclass algorithms such as the decision-tree algorithms C4.5 and CART, application of binary concept learning algorithms to learn individual binary functions for each of the k classes, and application of binary concept learning algorithms with distributed output representations. This paper compares these three approaches to a new technique in which error-correcting codes are employed as a distributed output representation. We show that these output representations improve the generalization performance of both C4.5 and backpropagation on a wide range of multiclass learning tasks. We also demonstrate that this approach is robust with respect to changes in the size of the training sample, the assignment of distributed representations to particular classes, and the application of overfitting avoidance techniques such as decision-tree pruning. Finally, we show that--like the other methods--the error-correcting code technique can provide reliable class probability estimates. Taken together, these results demonstrate that error-correcting output codes provide a general-purpose method for improving the performance of inductive learning programs on multiclass problems.
---
paper_title: The supervised learning no-free-lunch Theorems

paper_content:

This paper reviews the supervised learning versions of the no-free-lunch theorems in a simplified form. It also discusses the significance of those theorems, and their relation to other aspects of supervised learning.
---
paper_title: Soft computing methods applied to combination of one-class classifiers

paper_content:

The paper shows the possibilities of generalizing the two-class classification into multi-class classification by means of a fuzzy inference system. Fuzzy combiner harnesses the support values from classifiers to provide final response having no other restrictions on their structure. We compare proposed combination methods with ECOC and two variations of decision templates, based on Euclidean and symmetric distance. The effectiveness of the proposed combination method based on the fuzzy logic theory is also evaluated via computer experiments carried out on benchmark datasets.
---
paper_title: Trainable fusion rules. I. Large sample size case

paper_content:

A wide selection of standard statistical pattern classification algorithms can be applied as trainable fusion rules while designing neural network ensembles. A focus of the present two-part paper is finite sample effects: the complexity of base classifiers and fusion rules; the type of outputs provided by experts to the fusion rule; non-linearity of the fusion rule; degradation of experts and the fusion rule due to the lack of information in the design set; the adaptation of base classifiers to training set size, etc. In the first part of this paper, we consider arguments for utilizing continuous outputs of base classifiers versus categorical outputs and conclude: if one succeeds in having a small number of expert networks working perfectly in different parts of the input feature space, then crisp outputs may be preferable over continuous outputs. Afterwards, we oppose fixed fusion rules versus trainable ones and demonstrate situations where weighted average fusion can outperform simple average fusion. We present a review of statistical classification rules, paying special attention to these linear and non-linear rules, which are employed rarely but, according to our opinion, could be useful in neural network ensembles. We consider ideal and sample-based oracle decision rules and illustrate characteristic features of diverse fusion rules by considering an artificial two-dimensional (2D) example where the base classifiers perform well in different regions of input feature space.
---
paper_title: Combination of multiple classifiers using local accuracy estimates

paper_content:

This paper presents a method for combining classifiers that uses estimates of each individual classifier's local accuracy in small regions of feature space surrounding an unknown test sample. An empirical evaluation using five real data sets confirms the validity of our approach compared to some other combination of multiple classifiers algorithms. We also suggest a methodology for determining the best mix of individual classifiers.
---
paper_title: Optimizing group judgmental accuracy in the presence of interdependencies

paper_content:

Consider a group of people confronted with a dichotomous choice (for example, a yes or no decision). Assume that we can characterize each person by a probability, pi, of making the 'better' of the two choices open to the group, such that we define 'better' in terms of some linear ordering of the alternatives. If individual choices are independent, and if the a priori likelihood that either of the two choices is correct is one half, we show that the group decision procedure that maximizes the likelihood that the group will make the better of the two choices open to it is a weighted voting rule that assigns weights, wi, such that
---
paper_title: Trainable fusion rules. I. Large sample size case

paper_content:

A wide selection of standard statistical pattern classification algorithms can be applied as trainable fusion rules while designing neural network ensembles. A focus of the present two-part paper is finite sample effects: the complexity of base classifiers and fusion rules; the type of outputs provided by experts to the fusion rule; non-linearity of the fusion rule; degradation of experts and the fusion rule due to the lack of information in the design set; the adaptation of base classifiers to training set size, etc. In the first part of this paper, we consider arguments for utilizing continuous outputs of base classifiers versus categorical outputs and conclude: if one succeeds in having a small number of expert networks working perfectly in different parts of the input feature space, then crisp outputs may be preferable over continuous outputs. Afterwards, we oppose fixed fusion rules versus trainable ones and demonstrate situations where weighted average fusion can outperform simple average fusion. We present a review of statistical classification rules, paying special attention to these linear and non-linear rules, which are employed rarely but, according to our opinion, could be useful in neural network ensembles. We consider ideal and sample-based oracle decision rules and illustrate characteristic features of diverse fusion rules by considering an artificial two-dimensional (2D) example where the base classifiers perform well in different regions of input feature space.
---
paper_title: Limits on the majority vote accuracy in classifier fusion

paper_content:

We derive upper and lower limits on the majority vote accuracy with respect to individual accuracy p, the number of classiers in the pool (L), and the pairwise dependence between classiers, measured by Yule's Q statistic. Independence between individual classi- ers is typically viewed as an asset in classier fusion. We show that the majority vote with dependent classiers can potentially oer a dramatic improvement both over independent clas- siers and over the individual accuracy p. A functional relationship between the limits and the pairwise dependence Q is derived. Two patterns of the joint distribution for classier outputs (correct/incorrect) are identied to derive the limits: the pattern of success and the pattern of failure. The results support the intuition that negative pairwise dependence is benecial although not straightforwardly related to the accuracy. The pattern of success showed that for the highest improvement over p, all pairs of classiers in the pool should have the same negative dependence.
---
paper_title: An overview and comparison of voting methods for pattern recognition

paper_content:

In pattern recognition, there is a growing use of multiple classifier combinations with the goal to increase recognition performance. In many cases, plurality voting is a part of the combination process. In this article, we discuss and test several well known voting methods from politics and economics on classifier combination in order to see if an alternative to the simple plurality vote exists. We found that, assuming a number of prerequisites, better methods are available, that are comparatively simple and fast.
---
paper_title: Neural network ensembles

paper_content:

Several means for improving the performance and training of neural networks for classification are proposed. Crossvalidation is used as a tool for optimizing network parameters and architecture. It is shown that the remaining residual generalization error can be reduced by invoking ensembles of similar networks. >
---
paper_title: Trainable fusion rules. II. Small sample-size effects

paper_content:

Profound theoretical analysis is performed of small-sample properties of trainable fusion rules to determine in which situations neural network ensembles can improve or degrade classification results. We consider small sample effects, specific only to multiple classifiers system design in the two-category case of two important fusion rules: (1) linear weighted average (weighted voting), realized either by the standard Fisher classifier or by the single-layer perceptron, and (2) the non-linear Behavior-Knowledge-Space method. The small sample effects include: (i) training bias, i.e. learning sample size influence on generalization error of the base experts or of the fusion rule, (ii) optimistic biased outputs of the experts (self-boasting effect) and (iii) sample size impact on determining optimal complexity of the fusion rule. Correction terms developed to reduce the self-boasting effect are studied. It is shown that small learning sets increase classification error of the expert classifiers and damage correlation structure between their outputs. If the sizes of learning sets used to develop the expert classifiers are too small, non-trainable fusion rules can outperform more sophisticated trainable ones. A practical technique to fight sample size problems is a noise injection technique. The noise injection reduces the fusion rule's complexity and diminishes the expert's boasting bias.
---
paper_title: Some Remarks on Chosen Methods of Classifier Fusion Based on Weighted Voting

paper_content:

Multiple Classifier Systems are nowadays one of the most promising directions in pattern recognition. There are many methods of decision making by the ensemble of classifiers. The most popular are methods that have their origin in voting method, where the decision of the common classifier is a combination of individual classifiers' outputs. This work presents comparative analysis of some classifier fusion methods based on weighted voting of classifiers' responses and combination of classifiers' discriminant functions. We discus which of presented methods could produce classifier better than Oracle one. Some results of computer experiments carried out on benchmark and computer generated data which confirmed our studies are presented also.
---
paper_title: Combining Pattern Classifiers: Methods and Algorithms

paper_content:

This book, which is wholly devoted to the subject of model combination, is divided into ten chapters. In addition to the first two introductory chapters, the book covers some of the following topics: multiple classifier systems; combination methods when the base classifier outputs are 0/1; methods when the outputs are continuous, e.g., posterior probabilities; methods for classifier selection; bagging and boosting; the theory of fixed combination rules; and the concept of diversity. Overall, it is a very well-written monograph. It explains and analyzes different approaches comparatively so that the reader can see how they are similar and how they differ. The literature survey is extensive. The MATLAB code for many methods is given in chapter appendices allowing readers to play with the explained methods or apply them quickly to their own data. The book is a must-read for researchers and practitioners alike.
---
paper_title: Methods of combining multiple classifiers and their applications to handwriting recognition

paper_content:

Possible solutions to the problem of combining classifiers can be divided into three categories according to the levels of information available from the various classifiers. Four approaches based on different methodologies are proposed for solving this problem. One is suitable for combining individual classifiers such as Bayesian, k-nearest-neighbor, and various distance classifiers. The other three could be used for combining any kind of individual classifiers. On applying these methods to combine several classifiers for recognizing totally unconstrained handwritten numerals, the experimental results show that the performance of individual classifiers can be improved significantly. For example, on the US zipcode database, 98.9% recognition with 0.90% substitution and 0.2% rejection can be obtained, as well as high reliability with 95% recognition, 0% substitution, and 5% rejection. >
---
paper_title: Optimizing a multiple classifier system

paper_content:

Recently, multiple classifier systems (MCS) have been used for practical applications to improve classification accuracy. Self-generating neural networks (SGNN) are one of the suitable base-classifiers for MCS because of their simple setting and fast learning. However, the computation cost of the MCS increases in proportion to the number of SGNN. In this paper, we propose a novel optimization method for the structure of the SGNN in the MCS. We compare the optimized MCS with two sampling methods. Experiments have been conducted to compare the optimized MCS with an unoptimized MCS, the MCS based on C4.5, and k-nearest neighbor. The results show that the optimized MCS can improve its classification accuracy as well as reducing the computation cost.
---
paper_title: Adaptive Mixtures of Local Experts

paper_content:

We present a new supervised learning procedure for systems composed of many separate networks, each of which learns to handle a subset of the complete set of training cases. The new procedure can be viewed either as a modular version of a multilayer supervised network, or as an associative version of competitive learning. It therefore provides a new link between these two apparently different approaches. We demonstrate that the learning procedure divides up a vowel discrimination task into appropriate subtasks, each of which can be solved by a very simple expert network.
---
paper_title: A theoretical and experimental analysis of linear combiners for multiple classifier systems

paper_content:

In this paper, a theoretical and experimental analysis of linear combiners for multiple classifier systems is presented. Although linear combiners are the most frequently used combining rules, many important issues related to their operation for pattern classification tasks lack a theoretical basis. After a critical review of the framework developed in works by Turner and Ghosh [1996], [1999] on which our analysis is based, we focus on the simplest and most widely used implementation of linear combiners, which consists of assigning a nonnegative weight to each individual classifier. Moreover, we consider the ideal performance of this combining rule, i.e., that achievable when the optimal values of the weights are used. We do not consider the problem of weights estimation, which has been addressed in the literature. Our theoretical analysis shows how the performance of linear combiners, in terms of misclassification probability, depends on the performance of individual classifiers, and on the correlation between their outputs. In particular, we evaluate the ideal performance improvement that can be achieved using the weighted average over the simple average combining rule and investigate in what way it depends on the individual classifiers. Experimental results on real data sets show that the behavior of linear combiners agrees with the predictions of our analytical model. Finally, we discuss the contribution to the state of the art and the practical relevance of our theoretical and experimental analysis of linear combiners for multiple classifier systems.
---
paper_title: Combining Estimators Using Non-Constant Weighting Functions

paper_content:

This paper discusses the linearly weighted combination of estimators in which the weighting functions are dependent on the input. We show that the weighting functions can be derived either by evaluating the input dependent variance of each estimator or by estimating how likely it is that a given estimator has seen data in the region of the input space close to the input pattern. The latter solution is closely related to the mixture of experts approach and we show how learning rules for the mixture of experts can be derived from the theory about learning with missing features. The presented approaches are modular since the weighting functions can easily be modified (no retraining) if more estimators are added. Furthermore, it is easy to incorporate estimators which were not derived from data such as expert systems or algorithms.
---
paper_title: Bayesian analysis of linear combiners

paper_content:

A new theoretical framework for the analysis of linear combiners is presented in this paper. This framework extends the scope of previous analytical models, and provides some new theoretical results which improve the understanding of linear combiners operation. In particular, we show that the analytical model developed in seminal works by Tumer and Ghosh is included in this framework.
---
paper_title: Feature set decomposition for decision trees

paper_content:

This paper presents practical aspects of feature set decomposition in classification problems using decision trees. Feature set decomposition generalizes the task of feature selection which is extensively used in data mining. Feature selection aims to provide a representative set of features from which a classifier is constructed. On the other hand, feature set decomposition decomposes the original set of features into several subsets, and builds a classifier for each subset. The classifiers are then combined for classifying new instances. In order to examine the idea, a general framework that searches for helpful decomposition structures is proposed. This framework nests many algorithms, two of which are tested empirically over a set of benchmark datasets. The first algorithm performs a serial search while using a new Vapnik-Chervonenkis dimension bound for multiple oblivious trees as an evaluating schema. The second algorithm performs a multi-search while using wrapper evaluating schema. This work indicates that feature set decomposition can increase the accuracy of decision trees.
---
paper_title: A generic sensor fusion problem: Classification and function estimation

paper_content:

A generic fusion problem is studied for multiple sensors whose outputs are probabilistically related to their inputs according to unknown distributions. Sensor measurements are provided as iid input-output samples, and an empirical risk minimization method is described for designing fusers with distribution-free performance bounds. The special cases of isolation and projective fusers for classifiers and function estimators, respectively, are described in terms of performance bounds. The isolation fusers for classifiers are probabilistically guaranteed to perform at least as good as the best classifier. The projective fusers for function estimators are probabilistically guaranteed to perform at least as good as the best subset of estimators.
---
paper_title: Generating Accurate and Diverse Members of a Neural-Network Ensemble

paper_content:

Neural-network ensembles have been shown to be very accurate classification techniques. Previous work has shown that an effective ensemble should consist of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well. Most existing techniques, however, only indirectly address the problem of creating such a set of networks. In this paper we present a technique called ADDEMUP that uses genetic algorithms to directly search for an accurate and diverse set of trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to continually create new networks, keeping the set of networks that are as accurate as possible while disagreeing with each other as much as possible. Experiments on three DNA problems show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing approaches. Experiments also show that ADDEMUP is able to effectively incorporate prior knowledge, if available, to improve the quality of its ensemble.
---
paper_title: Combining independent and unbiased classifiers using weighted average

paper_content:

In a classification problem, improved accuracy can be obtained in many situations by using the combination of several classifiers instead of a single one. Turner and Gosh (1999) derived the error reduction that can be obtained by combining unbiased classifiers with independent errors using a simple average. We present an extension of this result by finding the improvement obtained when combining classifiers using weighted average. We also prove that for unbiased classifiers with independent errors the best combination of N classifiers corresponds to a weighted average, where the combination coefficient of each classifier is equal to 1/N. This means that in these cases the simple average should be used. We present experiments illustrating our results.
---
paper_title: Some Remarks on Chosen Methods of Classifier Fusion Based on Weighted Voting

paper_content:

Multiple Classifier Systems are nowadays one of the most promising directions in pattern recognition. There are many methods of decision making by the ensemble of classifiers. The most popular are methods that have their origin in voting method, where the decision of the common classifier is a combination of individual classifiers' outputs. This work presents comparative analysis of some classifier fusion methods based on weighted voting of classifiers' responses and combination of classifiers' discriminant functions. We discus which of presented methods could produce classifier better than Oracle one. Some results of computer experiments carried out on benchmark and computer generated data which confirmed our studies are presented also.
---
paper_title: Sum versus vote fusion in multiple classifier systems

paper_content:

Amidst the conflicting experimental evidence of superiority of one over the other, we investigate the Sum and majority Vote combining rules in a two class case, under the assumption of experts being of equal strength and estimation errors conditionally independent and identically distributed. We show, analytically, that, for Gaussian estimation error distributions, Sum always outperforms Vote. For heavy tail distributions, we demonstrate by simulation that Vote may outperform Sum. Results on synthetic data confirm the theoretical predictions. Experiments on real data support the general findings, but also show the effect of the usual assumptions of conditional independence, identical error distributions, and common target outputs of the experts not being fully satisfied.
---
paper_title: Optimal Linear Combinations Of Neural Networks

paper_content:

Neural network-based modeling often involves trying multiple networks with different architectures and training parameters in order to achieve acceptable model accuracy. Typically, one of the trained networks is chosen as best, while the rest are discarded. [Hashem and Schmeiser (1995)] proposed using optimal linear combinations of a number of trained neural networks instead of using a single best network. Combining the trained networks may help integrate the knowledge acquired by the components networks and thus improve model accuracy. In this paper, we extend the idea of optimal linear combinations (OLCs) of neural networks and discuss issues related to the generalization ability of the combined model. We then present two algorithms for selecting the component networks for the combination to improve the generalization ability of OLCs. Our experimental results demonstrate significant improvements in model accuracy, as a result of using OLCs, compared to using the apparent best network. Copyright 1997 Elsevier Science Ltd.
---
paper_title: Decision Combination in Multiple Classifier Systems

paper_content:

A multiple classifier system is a powerful solution to difficult pattern recognition problems involving large class sets and noisy input because it allows simultaneous use of arbitrary feature descriptors and classification procedures. Decisions by the classifiers can be represented as rankings of classifiers and different instances of a problem. The rankings can be combined by methods that either reduce or rerank a given set of classes. An intersection method and union method are proposed for class set reduction. Three methods based on the highest rank, the Borda count, and logistic regression are proposed for class set reranking. These methods have been tested in applications of degraded machine-printed characters and works from large lexicons, resulting in substantial improvement in overall correctness. >
---
paper_title: The combining classifier: to train or not to train?

paper_content:

When more than a single classifier has been trained for the same recognition problem the question arises how this set of classifiers may be combined into a final decision rule. Several fixed combining rules are used that depend on the output values of the base classifiers only. They are almost always suboptimal. Usually, however, training sets are available. They may be used to calibrate the base classifier outputs, as well as to build a trained combining classifier using these outputs as inputs. It depends on various circumstances whether this is useful, in particular whether the training set is used for the base classifiers as well and whether they are overtrained. We present an intuitive discussion on the use of trained combiners, relating the question of the choice of the combining classifier to a similar choice in the area of dissimilarity based pattern recognition. Some simple examples are used to illustrate the discussion.
---
paper_title: A method of combining multiple experts for the recognition of unconstrained handwritten numerals

paper_content:

For pattern recognition, when a single classifier cannot provide a decision which is 100 percent correct, multiple classifiers should be able to achieve higher accuracy. This is because group decisions are generally better than any individual's. Based on this concept, a method called the "Behavior-Knowledge Space Method" was developed, which can aggregate the decisions obtained from individual classifiers and derive the best final decisions from the statistical point of view. Experiments on 46451 samples of unconstrained handwritten numerals have shown that this method achieves very promising performances and outperforms voting, Bayesian, and Dempster-Shafer approaches. >
---
paper_title: Stacked Generalization

paper_content:

This paper introduces stacked generalization, a scheme for minimizing the generalization error rate of one or more generalizers. Stacked generalization works by deducing the biases of the generalizer(s) with respect to a provided learning set. This deduction proceeds by generalizing in a second space whose inputs are (for example) the guesses of the original generalizers when taught with part of the learning set and trying to guess the rest of it, and whose output is (for example) the correct guess. When used with multiple generalizers, stacked generalization can be seen as a more sophisticated version of cross-validation, exploiting a strategy more sophisticated than cross-validation's crude winner-takes-all for combining the individual generalizers. When used with a single generalizer, stacked generalization is a scheme for estimating (and then correcting for) the error of a generalizer which has been trained on a particular learning set and then asked a particular question. After introducing stacked generalization and justifying its use, this paper presents two numerical experiments. The first demonstrates how stacked generalization improves upon a set of separate generalizers for the NETtalk task of translating text to phonemes. The second demonstrates how stacked generalization improves the performance of a single surface-fitter. With the other experimental evidence in the literature, the usual arguments supporting cross-validation, and the abstract justifications presented in this paper, the conclusion is that for almost any real-world generalization problem one should use some version of stacked generalization to minimize the generalization error rate. This paper ends by discussing some of the variations of stacked generalization, and how it touches on other fields like chaos theory.
---
paper_title: Classifier selection for majority voting

paper_content:

Abstract Individual classification models are recently challenged by combined pattern recognition systems, which often show better performance. In such systems the optimal set of classifiers is first selected and then combined by a specific fusion method. For a small number of classifiers optimal ensembles can be found exhaustively, but the burden of exponential complexity of such search limits its practical applicability for larger systems. As a result, simpler search algorithms and/or selection criteria are needed to reduce the complexity. This work provides a revision of the classifier selection methodology and evaluates the practical applicability of diversity measures in the context of combining classifiers by majority voting. A number of search algorithms are proposed and adjusted to work properly with a number of selection criteria including majority voting error and various diversity measures. Extensive experiments carried out with 15 classifiers on 27 datasets indicate inappropriateness of diversity measures used as selection criteria in favour of the direct combiner error based search. Furthermore, the results prompted a novel design of multiple classifier systems in which selection and fusion are recurrently applied to a population of best combinations of classifiers rather than the individual best. The improvement of the generalisation performance of such system is demonstrated experimentally.
---
paper_title: Combining multiple classifiers based on statistical method for handwritten Chinese character recognition

paper_content:

In various application areas of pattern recognition, combining multiple classifiers is regarded as a method for achieving a substantial gain in performance of systems. The paper presents a method for handwritten Chinese character recognition to combine multiple classifiers based on statistics. Fusion strategies are discussed for providing a basis for combining classifiers. These combination strategies are experimentally tested on an online handwritten Chinese character recognition system. In our experiments, other combination approaches are also involved for comparison.
---
paper_title: Generating Accurate and Diverse Members of a Neural-Network Ensemble

paper_content:

Neural-network ensembles have been shown to be very accurate classification techniques. Previous work has shown that an effective ensemble should consist of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well. Most existing techniques, however, only indirectly address the problem of creating such a set of networks. In this paper we present a technique called ADDEMUP that uses genetic algorithms to directly search for an accurate and diverse set of trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to continually create new networks, keeping the set of networks that are as accurate as possible while disagreeing with each other as much as possible. Experiments on three DNA problems show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing approaches. Experiments also show that ADDEMUP is able to effectively incorporate prior knowledge, if available, to improve the quality of its ensemble.
---
paper_title: Multiple binary decision tree classifiers

paper_content:

Abstract Binary decision trees based on nonparametric statistical models of the data provide a solution to difficult decision problems where there are many classes and many available features related in a complex manner. Unfortunately, the technique requires a very large training set and is often limited by the size of the training set rather than by the discriminatory power of the features. This paper demonstrates that higher classification accuracies can be obtained from the same training set by using a combination of decision trees and by reaching a consensus using Dempster and Shafer's theory of evidence.
---
paper_title: Experiments with Trained and Untrained Fusers

paper_content:

The Multiple Classifier Systems are nowadays one of the most promising directions in pattern recognition. There are many methods of decision making by the ensemble of classifiers. The most popular are methods that have their origin in vote method, where the decision of the common classifier is a combination of individual classifiers decisions. This work presents method of weighted classifiers combination and experimental results of proposed algorithm.
---
paper_title: Optimal Linear Combinations Of Neural Networks

paper_content:

Neural network-based modeling often involves trying multiple networks with different architectures and training parameters in order to achieve acceptable model accuracy. Typically, one of the trained networks is chosen as best, while the rest are discarded. [Hashem and Schmeiser (1995)] proposed using optimal linear combinations of a number of trained neural networks instead of using a single best network. Combining the trained networks may help integrate the knowledge acquired by the components networks and thus improve model accuracy. In this paper, we extend the idea of optimal linear combinations (OLCs) of neural networks and discuss issues related to the generalization ability of the combined model. We then present two algorithms for selecting the component networks for the combination to improve the generalization ability of OLCs. Our experimental results demonstrate significant improvements in model accuracy, as a result of using OLCs, compared to using the apparent best network. Copyright 1997 Elsevier Science Ltd.
---
paper_title: Evolutionary approach to produce classifier ensemble based on weighted voting

paper_content:

Combining classifiers are nowadays one of the most promising direction in pattern recognition. There are many methods of decision making which could be used by the ensemble of classifiers. The most popular are methods that have their origin in voting, where the decision of the common classifier is a combination of individual classifiers' outputs, i.e. classifiers' responses (class numbers) or values of discriminants. This work presents classifiers' weights establish via evolutionary approach. We propose such method for classifier ensemble which could produce classifiers better than Oracle classifier what was shown in the previous works of author.
---
paper_title: Optimizing a multiple classifier system

paper_content:

Recently, multiple classifier systems (MCS) have been used for practical applications to improve classification accuracy. Self-generating neural networks (SGNN) are one of the suitable base-classifiers for MCS because of their simple setting and fast learning. However, the computation cost of the MCS increases in proportion to the number of SGNN. In this paper, we propose a novel optimization method for the structure of the SGNN in the MCS. We compare the optimized MCS with two sampling methods. Experiments have been conducted to compare the optimized MCS with an unoptimized MCS, the MCS based on C4.5, and k-nearest neighbor. The results show that the optimized MCS can improve its classification accuracy as well as reducing the computation cost.
---
paper_title: Constructing Ensembles from Data Envelopment Analysis

paper_content:

Using an ensemble of models often results in better performance than using a single “best” model. We present a new approach based on data envelopment analysis (DEA) for model combination. We prove that for two-class classification problems, DEA models identify the same convex hull as does the popular receiver operating characteristics (ROC) analysis used for model combination. We further develop two DEA-based methods to combine classifiers for the more general k-class classification problems. Our results demonstrate that the two methods outperform other benchmark methods and suggest that DEA can be a powerful tool for model combination.
---
paper_title: Classification of Customer Call Data in the Presence of Concept Drift and Noise

paper_content:

Many of today's real world domains require online classification tasks in very demanding situations. This work presents the results of applying the CD3 algorithm to telecommunications call data. CD3 enables the detection of concept drift in the presence of noise within real time data. The application detects the drift using a TSAR methodology and applies a purging mechanism as a corrective action. The main focus of this work is to identify from customer files and call records if the profile of customers registering for a 'friends and family' service is changing over a period of time. We will begin with a review of the CD3 application and the presentation of the data. This will conclude with experimental results.
---
paper_title: Change with Delayed Labeling: When is it Detectable?

paper_content:

Handling changes over time in supervised learning (concept drift) lately has received a great deal of attention, a number of adaptive learning strategies have been developed. Most of them make an optimistic assumption that the new labels become available immediately. In real sequential classification tasks it is often unrealistic due to task specific delayed labeling or associated labeling costs. We address the problem of change detectability, given, that the new labels are not available. In this analytical study we look at the space of changes from probabilistic perspective to analyze, what changes are detectable without seeing the labels and what are not. We conduct a range of experiments with real life data with simulated and natural changes to explore this detectability issue. We propose a computationally friendly detection technique, which monitors a stream of classifier outputs. We demonstrate analytically and experimentally, what types of changes are possible to detect when the labels for the new data are not available.
---
paper_title: Effective classification of noisy data streams with attribute-oriented dynamic classifier selection

paper_content:

Recently, mining from data streams has become an important and challenging task for many real-world applications such as credit card fraud protection and sensor networking. One popular solution is to separate stream data into chunks, learn a base classifier from each chunk, and then integrate all base classifiers for effective classification. In this paper, we propose a new dynamic classifier selection (DCS) mechanism to integrate base classifiers for effective mining from data streams. The proposed algorithm dynamically selects a single “best” classifier to classify each test instance at run time. Our scheme uses statistical information from attribute values, and uses each attribute to partition the evaluation set into disjoint subsets, followed by a procedure that evaluates the classification accuracy of each base classifier on these subsets. Given a test instance, its attribute values determine the subsets that the similar instances in the evaluation set have constructed, and the classifier with the highest classification accuracy on those subsets is selected to classify the test instance. Experimental results and comparative studies demonstrate the efficiency and efficacy of our method. Such a DCS scheme appears to be promising in mining data streams with dramatic concept drifting or with a significant amount of noise, where the base classifiers are likely conflictive or have low confidence.
---
paper_title: Mining data streams: a review

paper_content:

The recent advances in hardware and software have enabled the capture of different measurements of data in a wide range of fields. These measurements are generated continuously and in a very high fluctuating data rates. Examples include sensor networks, web logs, and computer network traffic. The storage, querying and mining of such data sets are highly computationally challenging tasks. Mining data streams is concerned with extracting knowledge structures represented in models and patterns in non stopping streams of information. The research in data stream mining has gained a high attraction due to the importance of its applications and the increasing generation of streaming information. Applications of data stream analysis can vary from critical scientific and astronomical applications to important business and financial ones. Algorithms, systems and frameworks that address streaming challenges have been developed over the past three years. In this review paper, we present the state-of-the-art in this growing vital field.
---
paper_title: An overview of anomaly detection techniques: Existing solutions and latest technological trends

paper_content:

As advances in networking technology help to connect the distant corners of the globe and as the Internet continues to expand its influence as a medium for communications and commerce, the threat from spammers, attackers and criminal enterprises has also grown accordingly. It is the prevalence of such threats that has made intrusion detection systems-the cyberspace's equivalent to the burglar alarm-join ranks with firewalls as one of the fundamental technologies for network security. However, today's commercially available intrusion detection systems are predominantly signature-based intrusion detection systems that are designed to detect known attacks by utilizing the signatures of those attacks. Such systems require frequent rule-base updates and signature updates, and are not capable of detecting unknown attacks. In contrast, anomaly detection systems, a subset of intrusion detection systems, model the normal system/network behavior which enables them to be extremely effective in finding and foiling both known as well as unknown or ''zero day'' attacks. While anomaly detection systems are attractive conceptually, a host of technological problems need to be overcome before they can be widely adopted. These problems include: high false alarm rate, failure to scale to gigabit speeds, etc. In this paper, we provide a comprehensive survey of anomaly detection systems and hybrid intrusion detection systems of the recent past and present. We also discuss recent technological trends in anomaly detection and identify open problems and challenges in this area.
---
paper_title: Mining concept-drifting data streams using ensemble classifiers

paper_content:

Recently, mining data streams with concept drifts for actionable insights has become an important and challenging task for a wide range of applications including credit card fraud protection, target marketing, network intrusion detection, etc. Conventional knowledge discovery tools are facing two challenges, the overwhelming volume of the streaming data, and the concept drifts. In this paper, we propose a general framework for mining concept-drifting data streams using weighted ensemble classifiers. We train an ensemble of classification models, such as C4.5, RIPPER, naive Beyesian, etc., from sequential chunks of the data stream. The classifiers in the ensemble are judiciously weighted based on their expected classification accuracy on the test data under the time-evolving environment. Thus, the ensemble approach improves both the efficiency in learning the model and the accuracy in performing classification. Our empirical study shows that the proposed methods have substantial advantage over single-classifier approaches in prediction accuracy, and the ensemble framework is effective for a variety of classification models.
---
paper_title: Classification of Changes in Evolving Data Streams using Online Clustering Result Deviation

paper_content:

Data stream analysis has attracted considerable attention over the past few years. Streaming data is often evolving over time. Capturing changes could be used for detecting an event or a phenomenon in various applications. Weather conditions, economical changes, astronomical and scientific phenomena are among a wide range of applications. Due to the high volume and speed of data streams, it is computationally hard to capture these changes from raw data in real-time. In this paper, we propose a novel algorithm that we term as STREAM-DETECT to capture these changes in data stream distribution and/or domain using clustering result deviation. STREAMDETECT is followed by a process of offline classification CHANGE-CLASS. This classification is concerned with the association of the history of change characteristics with the observed event or phenomenon. Experimental results show the efficiency of the proposed framework in both detecting the changes and classification accuracy.
---
paper_title: A streaming ensemble algorithm (SEA) for large-scale classification

paper_content:

Ensemble methods have recently garnered a great deal of attention in the machine learning community. Techniques such as Boosting and Bagging have proven to be highly effective but require repeated resampling of the training data, making them inappropriate in a data mining context. The methods presented in this paper take advantage of plentiful data, building separate classifiers on sequential chunks of training points. These classifiers are combined into a fixed-size ensemble using a heuristic replacement strategy. The result is a fast algorithm for large-scale or streaming data that classifies as well as a single decision tree built on all the data, requires approximately constant memory, and adjusts quickly to concept drift.
---
paper_title: Dynamic weighted majority: a new ensemble method for tracking concept drift

paper_content:

Algorithms for tracking concept drift are important for many applications. We present a general method based on the weighted majority algorithm for using any online learner for concept drift. Dynamic weighted majority (DWM) maintains an ensemble of base learners, predicts using a weighted-majority vote of these "experts", and dynamically creates and deletes experts in response to changes in performance. We empirically evaluated two experimental systems based on the method using incremental naive Bayes and incremental tree inducer [ITI] as experts. For the sake of comparison, we also included Blum's implementation of weighted majority. On the STAGGER concepts and on the SEA concepts, results suggest that the ensemble method learns drifting concepts almost as well as the base algorithms learn each concept individually. Indeed, we report the best overall results for these problems to date.
---
paper_title: Novelty detection: a review - part 1: statistical approaches

paper_content:

Novelty detection is the identification of new or unknown data or signal that a machine learning system is not aware of during training. Novelty detection is one of the fundamental requirements of a good classification or identification system since sometimes the test data contains information about objects that were not known at the time of training the model. In this paper we provide state-of-the-art review in the area of novelty detection based on statistical approaches. The second part paper details novelty detection using neural networks. As discussed, there are a multitude of applications where novelty detection is extremely important including signal processing, computer vision, pattern recognition, data mining, and robotics.
---
paper_title: Classifier Ensembles: Select Real-World Applications

paper_content:

Broad classes of statistical classification algorithms have been developed and applied successfully to a wide range of real-world domains. In general, ensuring that the particular classification algorithm matches the properties of the data is crucial in providing results that meet the needs of the particular application domain. One way in which the impact of this algorithm/application match can be alleviated is by using ensembles of classifiers, where a variety of classifiers (either different types of classifiers or different instantiations of the same classifier) are pooled before a final classification decision is made. Intuitively, classifier ensembles allow the different needs of a difficult problem to be handled by classifiers suited to those particular needs. Mathematically, classifier ensembles provide an extra degree of freedom in the classical bias/variance tradeoff, allowing solutions that would be difficult (if not impossible) to reach with only a single classifier. Because of these advantages, classifier ensembles have been applied to many difficult real-world problems. In this paper, we survey select applications of ensemble methods to problems that have historically been most representative of the difficulties in classification. In particular, we survey applications of ensemble methods to remote sensing, person recognition, one vs. all recognition, and medicine.
---
paper_title: Polarimetric classification of Boreal forest using nonparametric feature selection and multiple classifiers

paper_content:

Abstract Polarimetric SAR data contains a large amount of potential information that may be used to characterize forested scenes. However, the large number of PolSAR parameters and discriminators cannot all be used in most classification problems. Some form of feature selection will improve classification results and improve the efficiency of the system. In addition, classification of PolSAR data may be improved with an ensemble of classifiers, each tuned to a different class. Our research is in the Petawawa experimental forest, in the boreal forest northwest of Ottawa, Ontario, Canada. We employ Radarsat-2 fine-quad image data acquired in August (leaf-on) and November (leaf-off) of 2009. We present two system designs in this paper. The first system consists of a feature selector based on a non-parametric evaluation function and a support vector machine for classification. We demonstrate that the feature selection step improves classification accuracy significantly over a baseline classifier. We then present a system consisting of an ensemble of SVM classifiers, each with its own feature selection component and trained on an individual class. The classifier likelihoods are combined in a final step. We demonstrate that this system improves classification accuracy significantly over a single-classifier system. Finally, we demonstrate that classification accuracies are significantly higher when leaf-on and leaf-off images are combined over a single season image.
---
paper_title: Classification of multispectral images in coral environments using a hybrid of classifier ensembles

paper_content:

The accuracy attained in the mapping of underwater areas is limited by the effect of variations in the water column, which degrade the signal received by the orbital sensor, creating interclasses confusion that introduce errors into the final result of the classification process. In this article we will describe a hybrid classifier ensembles; the classification is done by progressive refining in three stages. At the end of this process, a combining unit links the various partial classifications generated and achieve the accuracy level desired. At the end, the result obtained by the ensemble is compared to the results achieved by the application of multi-class voting scheme methods based on support vector machine: One-Against-the-Rest and One-Against-One. Classification accuracy showed the viability and the potential of using the proposed ensemble to classify images.
---
paper_title: imageRF - A user-oriented implementation for remote sensing image analysis with Random Forests

paper_content:

An IDL implementation for the classification and regression analysis of remote sensing images with Random Forests is introduced. The tool, called imageRF, is platform and license independent and uses generic image file formats. It works well with default parameterization, yet all relevant parameters can be defined in intuitive GUIs. This makes it a user-friendly image processing tool, which is implemented as an add-on in the free EnMAP-Box and may be used in the commercial IDL/ENVI software.
---
paper_title: Uncertainty propagation in vegetation distribution models based on ensemble classifiers.

paper_content:

Ensemble learning techniques are increasingly applied for species and vegetation distribution modelling, often resulting in more accurate predictions. At the same time, uncertainty assessment of distribution models is gaining attention. In this study, Random Forests, an ensemble learning technique, is selected for vegetation distribution modelling based on environmental variables. The impact of two important sources of uncertainty, that is the uncertainty on spatial interpolation of environmental variables and the uncertainty on species clustering into vegetation types, is quantified based on sequential Gaussian simulation and pseudo-randomization tests, respectively. An empirical assessment of the uncertainty propagation to the distribution modelling results indicated a gradual decrease in performance with increasing input uncertainty. The test set error ranged from 30.83% to 52.63% and from 30.83% to 83.62%, when the uncertainty ranges on spatial interpolation and on vegetation clustering, respectively, were fully covered. Shannon’s entropy, which is proposed as a measure for uncertainty of ensemble predictions, revealed a similar increasing trend in prediction uncertainty. The implications of these results in an empirical distribution modelling framework are further discussed with respect to monitoring setup, spatial interpolation and species clustering.
---
paper_title: A self-trained ensemble with semisupervised SVM: An application to pixel classification of remote sensing imagery

paper_content:

In this article, we present a semisupervised support vector machine that uses self-training approach. We then construct an ensemble of semisupervised SVM classifiers to address the problem of pixel classification of remote sensing images. Semisupervised support vector machines (S^3VMs) are based on applying the margin maximization principle to both labeled and unlabeled samples. The ensemble of SVM classifiers recognizes the conceptual similarity between component classifiers from the same data source. The effectiveness of the proposed technique is first demonstrated for two numeric remote sensing data described in terms of feature vectors and then identifying different land cover regions in remote sensing imagery. Experimental results on these datasets show that employing this learning scheme can increase the accuracy level. The performance of the ensemble is compared with one of its component classifier and conventional SVM in terms of accuracy and quantitative cluster validity indices.
---
paper_title: Classifier ensembles for land cover mapping using multitemporal SAR imagery

paper_content:

SAR data are almost independent from weather conditions, and thus are well suited for mapping of seasonally changing variables such as land cover. In regard to recent and upcoming missions, multitemporal and multi-frequency approaches become even more attractive. In the present study, classifier ensembles (i.e., boosted decision tree and random forests) are applied to multi-temporal C-band SAR data, from different study sites and years. A detailed accuracy assessment shows that classifier ensembles, in particularly random forests, outperform standard approaches like a single decision tree and a conventional maximum likelihood classifier by more than 10% independently from the site and year. They reach up to almost 84% of overall accuracy in rural areas with large plots. Visual interpretation confirms the statistical accuracy assessment and reveals that also typical random noise is considerably reduced. In addition the results demonstrate that random forests are less sensitive to the number of training samples and perform well even with only a small number. Random forests are computationally highly efficient and are hence considered very well suited for land cover classifications of future multifrequency and multitemporal stacks of SAR imagery.
---
paper_title: An assessment of the effectiveness of a random forest classifier for land-cover classification

paper_content:

Land cover monitoring using remotely sensed data requires robust classification methods which allow for the accurate mapping of complex land cover and land use categories. Random forest (RF) is a powerful machine learning classifier that is relatively unknown in land remote sensing and has not been evaluated thoroughly by the remote sensing community compared to more conventional pattern recognition techniques. Key advantages of RF include: their non-parametric nature; high classification accuracy; and capability to determine variable importance. However, the split rules for classification are unknown, therefore RF can be considered to be black box type classifier. RF provides an algorithm for estimating missing values; and flexibility to perform several types of data analysis, including regression, classification, survival analysis, and unsupervised learning. ::: ::: In this paper, the performance of the RF classifier for land cover classification of a complex area is explored. Evaluation was based on several criteria: mapping accuracy, sensitivity to data set size and noise. Landsat-5 Thematic Mapper data captured in European spring and summer were used with auxiliary variables derived from a digital terrain model to classify 14 different land categories in the south of Spain. Results show that the RF algorithm yields accurate land cover classifications, with 92% overall accuracy and a Kappa index of 0.92. RF is robust to training data reduction and noise because significant differences in kappa values were only observed for data reduction and noise addition values greater than 50 and 20%, respectively. Additionally, variables that RF identified as most important for classifying land cover coincided with expectations. A McNemar test indicates an overall better performance of the random forest model over a single decision tree at the 0.00001 significance level.
---
paper_title: Evaluation of Random Forest and Adaboost tree-based ensemble classification and spectral band selection for ecotope mapping using airborne hyperspectral imagery

paper_content:

Abstract Detailed land use/land cover classification at ecotope level is important for environmental evaluation. In this study, we investigate the possibility of using airborne hyperspectral imagery for the classification of ecotopes. In particular, we assess two tree-based ensemble classification algorithms: Adaboost and Random Forest, based on standard classification accuracy, training time and classification stability. Our results show that Adaboost and Random Forest attain almost the same overall accuracy (close to 70%) with less than 1% difference, and both outperform a neural network classifier (63.7%). Random Forest, however, is faster in training and more stable. Both ensemble classifiers are considered effective in dealing with hyperspectral data. Furthermore, two feature selection methods, the out-of-bag strategy and a wrapper approach feature subset selection using the best-first search method are applied. A majority of bands chosen by both methods concentrate between 1.4 and 1.8 μm at the early shortwave infrared region. Our band subset analyses also include the 22 optimal bands between 0.4 and 2.5 μm suggested in Thenkabail et al. [Thenkabail, P.S., Enclona, E.A., Ashton, M.S., and Van Der Meer, B. (2004). Accuracy assessments of hyperspectral waveband performance for vegetation analysis applications. Remote Sensing of Environment, 91, 354–376.] due to similarity of the target classes. All of the three band subsets considered in this study work well with both classifiers as in most cases the overall accuracy dropped only by less than 1%. A subset of 53 bands is created by combining all feature subsets and comparing to using the entire set the overall accuracy is the same with Adaboost, and with Random Forest, a 0.2% improvement. The strategy to use a basket of band selection methods works better. Ecotopes belonging to the tree classes are in general classified better than the grass classes. Small adaptations of the classification scheme are recommended to improve the applicability of remote sensing method for detailed ecotope mapping.
---
paper_title: Design of effective multiple classifier systems by clustering of classifiers

paper_content:

In the field of pattern recognition, multiple classifier systems based on the combination of outputs of a set of different classifiers have been proposed as a method for the development of high performance classification systems. Previous work clearly showed that multiple classifier. Systems are effective only if the classifiers forming them make independent errors. Therefore, the fundamental need for methods aimed to design "error-independent" classifiers is currently acknowledged. In the paper, an approach to the automatic design of multiple classifier systems is proposed. Given an initial large set of classifiers, our approach is aimed at selecting the subset formed by the most error-independent classifiers. Reported results on the classification of multisensor remote-sensing images show that this approach allows to design effective multiple classifier systems.
---
paper_title: Remote sensing image classification based on neural network ensemble algorithm

paper_content:

The amounts and types of remote sensing data have increased rapidly, and the classification of these datasets has become more and more overwhelming for a single classifier in practical applications. In this paper, an ensemble algorithm based on Diversity Ensemble Creation by Oppositional Relabeling of Artificial Training Examples (DECORATEs) and Rotation Forest is proposed to solve the classification problem of remote sensing image. In this ensemble algorithm, the RBF neural networks are employed as base classifiers. Furthermore, interpolation technology for identical distribution is used to remold the input datasets. These remolded datasets will construct new classifiers besides the initial classifiers constructed by the Rotation Forest algorithm. The change of classification error is used to decide whether to add another new classifier. Therefore, the diversity among these classifiers will be enhanced and the accuracy of classification will be improved. Adaptability of the proposed algorithm is verified in experiments implemented on standard datasets and actual remote sensing dataset.
---
paper_title: Design of Effective Neural Network Ensembles for Image Classification Purposes

paper_content:

Abstract In the field of pattern recognition, the combination of an ensemble of neural networks has been proposed as an approach to the development of high performance image classification systems. However, previous work clearly showed that such image classification systems are effective only if the neural networks forming them make different errors. Therefore, the fundamental need for methods aimed to design ensembles of ‘error-independent’ networks is currently acknowledged. In this paper, an approach to the automatic design of effective neural network ensembles is proposed. Given an initial large set of neural networks, our approach is aimed to select the subset formed by the most error-independent nets. Reported results on the classification of multisensor remote-sensing images show that this approach allows one to design effective neural network ensembles.
---
paper_title: Random Forests for land cover classification

paper_content:

Random Forests are considered for classification of multisource remote sensing and geographic data. Various ensemble classification methods have been proposed in recent years. These methods have been proven to improve classification accuracy considerably. The most widely used ensemble methods are boosting and bagging. Boosting is based on sample re-weighting but bagging uses bootstrapping. The Random Forest classifier uses bagging, or bootstrap aggregating, to form an ensemble of classification and regression tree (CART)-like classifiers. In addition, it searches only a random subset of the variables for a split at each CART node, in order to minimize the correlation between the classifiers in the ensemble. This method is not sensitive to noise or overtraining, as the resampling is not based on weighting. Furthermore, it is computationally much lighter than methods based on boosting and somewhat lighter than simple bagging. In the paper, the use of the Random Forest classifier for land cover classification is explored. We compare the accuracy of the Random Forest classifier to other better-known ensemble methods on multisource remote sensing and geographic data.
---
paper_title: Combination of neural and statistical algorithms for supervised classification of remote-sensing images

paper_content:

Abstract Various experimental comparisons of algorithms for supervised classification of remote-sensing images have been reported in the literature. Among others, a comparison of neural and statistical classifiers has previously been made by the authors in (Serpico, S.B., Bruzzone, L., Roli, F., 1996. Pattern Recognition Letters 17, 1331–1341). Results of reported experiments have clearly shown that the superiority of one algorithm over another cannot be claimed. In addition, they have pointed out that statistical and neural algorithms often require expensive design phases to attain high classification accuracy. In this paper, the combination of neural and statistical algorithms is proposed as a method to obtain high accuracy values after much shorter design phases and to improve the accuracy–rejection tradeoff over those allowed by single algorithms.
---
paper_title: it COMBINING PARAMETRIC AND NON-PARAMETRIC ALGORITHMS FOR A PARTIALLY UNSUPERVISED CLASSIFICATION OF MULTITEMPORAL REMOTE-SENSING IMAGES

paper_content:

Abstract In this paper, we propose a classification system based on a multiple-classifier architecture, which is aimed at updating land-cover maps by using multisensor and/or multisource remote-sensing images. The proposed system is composed of an ensemble of classifiers that, once trained in a supervised way on a specific image of a given area, can be retrained in an unsupervised way to classify a new image of the considered site. In this context, two techniques are presented for the unsupervised updating of the parameters of a maximum-likelihood classifier and a radial basis function neural-network classifier, on the basis of the distribution of the new image to be classified. Experimental results carried out on a multitemporal and multisource remote-sensing data set confirm the effectiveness of the proposed system.
---
paper_title: Detection of land-cover transitions by combining multidate classifiers

paper_content:

This paper addresses the problem of detecting land-cover transitions by analysing multitemporal remote-sensing images. In order to develop an effective system for the detection of land-cover transitions, an ensemble of non-parametric multitemporal classifiers is defined and integrated in the context of a multiple classifier system (MCS). Each multitemporal classifier is developed in the framework of the compound classification (CC) decision rule. To develop as uncorrelated as possible classification procedures, the estimates of statistical parameters of classifiers are carried out according to different approaches (i.e., multilayer perceptron neural networks, radial basis functions neural networks, and k-nearest neighbour technique). The outputs provided by different classifiers are combined according to three standard strategies extended to the compound classification case (i.e., Majority voting, Bayesian average, and Bayesian weighted average). Experiments, carried out on a multitemporal remote-sensing data set, confirm the effectiveness of the proposed system.
---
paper_title: Information fusion techniques for change detection from multi-temporal remote sensing images

paper_content:

In order to investigate the impacts of different information fusion techniques on change detection, a sequential fusion strategy combining pan-sharpening with decision level fusion is introduced into change detection from multi-temporal remotely sensed images. Generally, change map from multi-temporal remote sensing images using any single method or single kind of data source may contain a number of omission/commission errors, degrading the detection accuracy to a great extent. To take advantage of the merits of multi-resolution image and multiple information fusion schemes, the proposed procedure consists of two steps: (1) change detection from pan-sharpened images, and (2) final change detection map generation by decision level fusion. Impacts of different fusion techniques on change detection results are evaluated by unsupervised similarity metric and supervised accuracy indices. Multi-temporal QuickBird and ALOS images are used for experiments. The experimental results demonstrate the positive impacts of different fusion strategies on change detection. Especially, pan-sharpening techniques improve spatial resolution and image quality, which effectively reduces the omission errors in change detection; and decision level fusion integrates the change maps from spatially enhanced fusion datasets and can well reduce the commission errors. Therefore, the overall accuracy of change detection can be increased step by step by the proposed sequential fusion framework.
---
paper_title: Detection of distributed denial of service attacks using an ensemble of adaptive and hybrid neuro-fuzzy systems

paper_content:

A DDoS attack is the most prevalent threat, viz., flooding the computing and communication resources in order to make the service unavailable for legitimate users, since a decade and continues to be threatening till date. Therefore, these critical resources must be protected against the DDoS attacks. The detection of DDoS attacks requires adaptive and incremental learning classifier, less computational complexity, and accurate decision making from uncertain information. Hence, the DDoS attacks could be detected using existing soft computing techniques such as fuzzy logic, neural networks, and genetic algorithms. Fuzzy logic has the advantage of interpreting the rules well but it suffers from the disadvantage of not able to acquire the rules automatically. The neural networks generalize the network well but they cannot interpret the rules. Genetic algorithm provides optimal solutions but the time complexity is high. Hybrid methods, Neuro-fuzzy and genetic fuzzy have been proposed to overcome the drawbacks of interpretability and manual rules acquisition. In this paper, adaptive and hybrid neuro-fuzzy systems were proposed as subsystems of the ensemble. Sugeno type Adaptive Neuro-Fuzzy Inference System (ANFIS) has been chosen as a base classifier for our research as Mamdani type ANFIS is not suitable for real time due to its high computational complexity and non-adaptiveness to extract exact knowledge from the dataset. Single classifier makes error on different training samples. So, by creating an ensemble of classifiers and combining their outputs, the total error can be reduced and the detection accuracy can be increased. Improvement in the performance of ANFIS ensemble is the focus of this paper. Our proposed DDoS classification algorithm, NFBoost, differs from the existing methods in weight update distribution strategy, error cost minimization, and ensemble output combination method, but resembles similar in classifier weight assignment and error computation. Our proposed NFBoost algorithm is achieved by combining ensemble of classifier outputs and Neyman Pearson cost minimization strategy, for final classification decision. Publicly available datasets such as KDD Cup, CAIDA DDOS Attack 2007, CONFICKER worm, UNINA traffic traces, and UCI Datasets were used for the simulation experiments. NFBoost was trained and tested with the publicly available datasets and our own SSE Lab SSENET 2011 datasets. Detection accuracy and Cost per sample were the two metrics used to analyze the performance of the NFBoost classification algorithm and were compared with bagging, boosting, and AdaBoost algorithms. From the simulation results, it is evident that NFBoost algorithm achieves high detection accuracy (99.2%) with fewer false alarms. Cost per instance is also very less for the NFBoost algorithm compared to the existing algorithms. NFBoost algorithm outperforms the existing ensemble algorithms with a maximum gain of 8.4% and a minimum gain of 1.1%.
---
paper_title: Distributed denial of service attack detection using an ensemble of neural classifier

paper_content:

The vulnerabilities in the Communication (TCP/IP) protocol stack and the availability of more sophisticated attack tools breed in more and more network hackers to attack the network intentionally or unintentionally, leading to Distributed Denial of Service (DDoS) attack. The DDoS attacks could be detected using the existing machine learning techniques such as neural classifiers. These classifiers lack generalization capabilities which result in less performance leading to high false positives. This paper evaluates the performance of a comprehensive set of machine learning algorithms for selecting the base classifier using the publicly available KDD Cup dataset. Based on the outcome of the experiments, Resilient Back Propagation (RBP) was chosen as base classifier for our research. The improvement in performance of the RBP classifier is the focus of this paper. Our proposed classification algorithm, RBPBoost, is achieved by combining ensemble of classifier outputs and Neyman Pearson cost minimization strategy, for final classification decision. Publicly available datasets such as KDD Cup, DARPA 1999, DARPA 2000, and CONFICKER were used for the simulation experiments. RBPBoost was trained and tested with DARPA, CONFICKER, and our own lab datasets. Detection accuracy and Cost per sample were the two metrics evaluated to analyze the performance of the RBPBoost classification algorithm. From the simulation results, it is evident that RBPBoost algorithm achieves high detection accuracy (99.4%) with fewer false alarms and outperforms the existing ensemble algorithms. RBPBoost algorithm outperforms the existing algorithms with maximum gain of 6.6% and minimum gain of 0.8%.
---
paper_title: Detection of malicious code by applying machine learning classifiers on static features: A state-of-the-art survey

paper_content:

This research synthesizes a taxonomy for classifying detection methods of new malicious code by Machine Learning (ML) methods based on static features extracted from executables. The taxonomy is then operationalized to classify research on this topic and pinpoint critical open research issues in light of emerging threats. The article addresses various facets of the detection challenge, including: file representation and feature selection methods, classification algorithms, weighting ensembles, as well as the imbalance problem, active learning, and chronological evaluation. From the survey we conclude that a framework for detecting new malicious code in executable files can be designed to achieve very high accuracy while maintaining low false positives (i.e. misclassifying benign files as malicious). The framework should include training of multiple classifiers on various types of features (mainly OpCode and byte n-grams and Portable Executable Features), applying weighting algorithm on the classification results of the individual classifiers, as well as an active learning mechanism to maintain high detection accuracy. The training of classifiers should also consider the imbalance problem by generating classifiers that will perform accurately in a real-life situation where the percentage of malicious files among all files is estimated to be approximately 10%.
---
paper_title: Modeling intrusion detection system using hybrid intelligent systems

paper_content:

The process of monitoring the events occurring in a computer system or network and analyzing them for sign of intrusions is known as intrusion detection system (IDS). This paper presents two hybrid approaches for modeling IDS. Decision trees (DT) and support vector machines (SVM) are combined as a hierarchical hybrid intelligent system model (DT-SVM) and an ensemble approach combining the base classifiers. The hybrid intrusion detection model combines the individual base classifiers and other hybrid machine learning paradigms to maximize detection accuracy and minimize computational complexity. Empirical results illustrate that the proposed hybrid systems provide more accurate intrusion detection systems.
---
paper_title: Anomalous Payload-Based Worm Detection and Signature Generation

paper_content:

New features of the PAYL anomalous payload detection sensor are demonstrated to accurately detect and generate signatures for zero-day worms. Experimental evidence demonstrates that site-specific packet content models are capable of detecting new worms with high accuracy in a collaborative security system. A new approach is proposed that correlates ingress/egress payload alerts to identify the worm's initial propagation. The method also enables automatic signature generation that can be deployed immediately to network firewalls and content filters to proactively protect other hosts. We also propose a collaborative privacy-preserving security strategy whereby different hosts can exchange PAYL signatures to increase accuracy and mitigate against false positives. The important principle demonstrated is that correlating multiple alerts identifies true positives from the set of anomaly alerts and reduces incorrect decisions producing accurate mitigation.
---
paper_title: Intrusion detection in computer networks by a modular ensemble of one-class classifiers

paper_content:

Since the early days of research on intrusion detection, anomaly-based approaches have been proposed to detect intrusion attempts. Attacks are detected as anomalies when compared to a model of normal (legitimate) events. Anomaly-based approaches typically produce a relatively large number of false alarms compared to signature-based IDS. However, anomaly-based IDS are able to detect never-before-seen attacks. As new types of attacks are generated at an increasing pace and the process of signature generation is slow, it turns out that signature-based IDS can be easily evaded by new attacks. The ability of anomaly-based IDS to detect attacks never observed in the wild has stirred up a renewed interest in anomaly detection. In particular, recent work focused on unsupervised or unlabeled anomaly detection, due to the fact that it is very hard and expensive to obtain a labeled dataset containing only pure normal events. The unlabeled approaches proposed so far for network IDS focused on modeling the normal network traffic considered as a whole. As network traffic related to different protocols or services exhibits different characteristics, this paper proposes an unlabeled Network Anomaly IDS based on a modular Multiple Classifier System (MCS). Each module is designed to model a particular group of similar protocols or network services. The use of a modular MCS allows the designer to choose a different model and decision threshold for different (groups of) network services. This also allows the designer to tune the false alarm rate and detection rate produced by each module to optimize the overall performance of the ensemble. Experimental results on the KDD-Cup 1999 dataset show that the proposed anomaly IDS achieves high attack detection rate and low false alarm rate at the same time.
---
paper_title: FLIPS: Hybrid Adaptive Intrusion Prevention

paper_content:

Intrusion detection systems are fundamentally passive and fail-open. Because their primary task is classification, they do nothing to prevent an attack from succeeding. An intrusion prevention system (IPS) adds protection mechanisms that provide fail-safe semantics, automatic response capabilities, and adaptive enforcement. We present FLIPS (Feedback Learning IPS), a hybrid approach to host security that prevents binary code injection attacks. It incorporates three major components: an anomaly-based classifier, a signature-based filtering scheme, and a supervision framework that employs Instruction Set Randomization (ISR). Since ISR prevents code injection attacks and can also precisely identify the injected code, we can tune the classifier and the filter via a learning mechanism based on this feedback. Capturing the injected code allows FLIPS to construct signatures for zero-day exploits. The filter can discard input that is anomalous or matches known malicious input, effectively protecting the application from additional instances of an attack - even zero-day attacks or attacks that are metamorphic in nature. FLIPS does not require a known user base and can be deployed transparently to clients and with minimal impact on servers. We describe a prototype that protects HTTP servers, but FLIPS can be applied to a variety of server and client applications.
---
paper_title: Ensemble based sensing anomaly detection in wireless sensor networks

paper_content:

Highlights? We developed an efficient ensemble of five classifiers to detect sensor anomalies. ? We model the correct sensor behavior in complex/dynamic environments using five models. ? We solve the problems related to training, testing and evaluation of the ensemble. ? Our ensemble offers reliable estimations to replace the erroneous measurements. ? We present a case study to show the preciseness of our method. Wireless sensor networks are often used to monitor and measure physical characteristics from remote and sometimes hostile environments. In these circumstances the sensing data accuracy is a crucial attribute for the way these applications complete their objectives, requiring efficient solutions to discover sensor anomalies. Such solutions are hard to be found mainly because the intricate defining of the correct sensor behavior in a complex and dynamic environment. This paper tackles the sensing anomaly detection from a new perspective by modeling the correct operation of sensors not by one, but by five different dynamical models, acting synergically to provide a reliable solution. Our methodology relies on an ensemble based system composed of a set of diverse binary classifiers, adequately selected, to implement a complex decisional system on network base station. Moreover, every time a sensing anomaly is discovered, our ensemble offers a reliable estimation to replace the erroneous measurement provided by sensor.
---
paper_title: Statistical fraud detection: A review

paper_content:

Fraud is increasing dramatically with the expansion of modem technology and the global superhighways of communication, resulting in the loss of billions of dollars worldwide each year. Although prevention technologies are the best way to reduce fraud, fraudsters are adaptive and, given time, will usually find ways to circumvent such measures. Methodologies for the detection of fraud are essential if we are to catch fraudsters once fraud prevention has failed. Statistics and machine learning provide effective technologies for fraud detection and have been applied successfully to detect activities such as money laundering, e-commerce credit card fraud, telecommunications fraud and computer intrusion, to name but a few. We describe the tools available for statistical fraud detection and the areas in which fraud detection technologies are most used.
---
paper_title: Bagging k-dependence probabilistic networks: An alternative powerful fraud detection tool

paper_content:

Fraud is a global problem that has required more attention due to an accentuated expansion of modern technology and communication. When statistical techniques are used to detect fraud, whether a fraud detection model is accurate enough in order to provide correct classification of the case as a fraudulent or legitimate is a critical factor. In this context, the concept of bootstrap aggregating (bagging) arises. The basic idea is to generate multiple classifiers by obtaining the predicted values from the adjusted models to several replicated datasets and then combining them into a single predictive classification in order to improve the classification accuracy. In this paper, for the first time, we aim to present a pioneer study of the performance of the discrete and continuous k-dependence probabilistic networks within the context of bagging predictors classification. Via a large simulation study and various real datasets, we discovered that the probabilistic networks are a strong modeling option with high predictive capacity and with a high increment using the bagging procedure when compared to traditional techniques.
---
paper_title: Support vector machine based multiagent ensemble learning for credit risk evaluation

paper_content:

In this paper, a four-stage support vector machine (SVM) based multiagent ensemble learning approach is proposed for credit risk evaluation. In the first stage, the initial dataset is divided into two independent subsets: training subset (in-sample data) and testing subset (out-of-sample data) for training and verification purposes. In the second stage, different SVM learning paradigms with much dissimilarity are constructed as intelligent agents for credit risk evaluation. In the third stage, multiple individual SVM agents are trained using training subsets and the corresponding evaluation results are also obtained. In the final stage, all individual results produced by multiple SVM agents in the previous stage are aggregated into an ensemble result. In particular, the impact of the diversity of individual intelligent agents on the generalization performance of the SVM-based multiagent ensemble learning system is examined and analyzed. For illustration, one corporate credit card application approval dataset is used to verify the effectiveness of the SVM-based multiagent ensemble learning system.
---
paper_title: Data mining for credit card fraud: A comparative study

paper_content:

Credit card fraud is a serious and growing problem. While predictive models for credit card fraud detection are in active use in practice, reported studies on the use of data mining approaches for credit card fraud detection are relatively few, possibly due to the lack of available data for research. This paper evaluates two advanced data mining approaches, support vector machines and random forests, together with the well-known logistic regression, as part of an attempt to better detect (and thus control and prosecute) credit card fraud. The study is based on real-life data of transactions from an international credit card operation.
---
paper_title: Stock fraud detection using peer group analysis

paper_content:

Highlights? We apply peer group analysis to the detection of stock price manipulation. ? We update different weights of peer group members over time. ? We analyze real time series data observed from Korean stock market. This study proposes a method to detect suspicious patterns of stock price manipulation using an unsupervised data mining technique: peer group analysis. This technique detects abnormal behavior of a target by comparing it with its peer group and measuring the deviation of its behavior from that of its peers. Moreover, this study proposes a method to improve the general peer group analysis by incorporating the weight of peer group members into summarizing their behavior, along with the consideration of parameter updates over time. Using real time series data of Korean stock market, this study shows the advantage of the proposed peer group analysis in detecting abnormal stock price change. In addition, we perform sensitivity analysis to examine the effect of the parameters used in the proposed method.
---
paper_title: Multiple classifier application to credit risk assessment

paper_content:

Credit risk prediction models seek to predict quality factors such as whether an individual will default (bad applicant) on a loan or not (good applicant). This can be treated as a kind of machine learning (ML) problem. Recently, the use of ML algorithms has proven to be of great practical value in solving a variety of risk problems including credit risk prediction. One of the most active areas of recent research in ML has been the use of ensemble (combining) classifiers. Research indicates that ensemble individual classifiers lead to a significant improvement in classification performance by having them vote for the most popular class. This paper explores the predicted behaviour of five classifiers for different types of noise in terms of credit risk prediction accuracy, and how such accuracy could be improved by using classifier ensembles. Benchmarking results on four credit datasets and comparison with the performance of each individual classifier on predictive accuracy at various attribute noise levels are presented. The experimental evaluation shows that the ensemble of classifiers technique has the potential to improve prediction accuracy.
---
paper_title: Detection of financial statement fraud and feature selection using data mining techniques

paper_content:

Recently, high profile cases of financial statement fraud have been dominating the news. This paper uses data mining techniques such as Multilayer Feed Forward Neural Network (MLFF), Support Vector Machines (SVM), Genetic Programming (GP), Group Method of Data Handling (GMDH), Logistic Regression (LR), and Probabilistic Neural Network (PNN) to identify companies that resort to financial statement fraud. Each of these techniques is tested on a dataset involving 202 Chinese companies and compared with and without feature selection. PNN outperformed all the techniques without feature selection, and GP and PNN outperformed others with feature selection and with marginally equal accuracies.
---
paper_title: A hybrid ensemble approach for enterprise credit risk assessment based on Support Vector Machine

paper_content:

Enterprise credit risk assessment has long been regarded as a critical topic and many statistical and intelligent methods have been explored for this issue. However there are no consistent conclusions on which methods are better. Recent researches suggest combining multiple classifiers, i.e., ensemble learning, may have a better performance. In this paper, we propose a new hybrid ensemble approach, called RSB-SVM, which is based on two popular ensemble strategies, i.e., bagging and random subspace and uses Support Vector Machine (SVM) as base learner. As there are two different factors, i.e., bootstrap selection of instances and random selection of features, encouraging diversity in RSB-SVM, it would be advantageous to get better performance. The enterprise credit risk dataset, which includes 239 companies' financial records and is collected by the Industrial and Commercial Bank of China, is selected to demonstrate the effectiveness and feasibility of proposed method. Experimental results reveal that RSB-SVM can be used as an alternative method for enterprise credit risk assessment.
---
paper_title: Classifiers selection in ensembles using genetic algorithms for bankruptcy prediction

paper_content:

Ensemble learning is a method to improve the performance of classification and prediction algorithms. Many studies have demonstrated that ensemble learning can decrease the generalization error and improve the performance of individual classifiers and predictors. However, its performance can be degraded due to multicollinearity problem where multiple classifiers of an ensemble are highly correlated with. This paper proposes a genetic algorithm-based coverage optimization technique in the purpose of resolving multicollinearity problem. Empirical results with bankruptcy prediction on Korea firms indicate that the proposed coverage optimization algorithm can help to design a diverse and highly accurate classification system.
---
paper_title: Multiple classifier architectures and their application to credit risk assessment

paper_content:

Multiple classifier systems combine several individual classifiers to deliver a final classification decision. In this paper the performance of several multiple classifier systems are evaluated in terms of their ability to correctly classify consumers as good or bad credit risks. Empirical results suggest that some multiple classifier systems deliver significantly better performance than the single best classifier, but many do not. Overall, bagging and boosting outperform other multi-classifier systems, and a new boosting algorithm, Error Trimmed Boosting, outperforms bagging and AdaBoost by a significant margin.
---
paper_title: Failure prediction of dotcom companies using neural network-genetic programming hybrids

paper_content:

This paper presents novel neural network-genetic programming hybrids to predict the failure of dotcom companies. These hybrids comprise multilayer feed forward neural network (MLFF), probabilistic neural network (PNN), rough sets (RS) and genetic programming (GP) in a two-phase architecture. In each hybrid, one technique is used to perform feature selection in the first phase and another one is used as a classifier in the second phase. Further t-statistic and f-statistic are also used separately for feature selection in the first phase. In each of these cases, top 10 features are selected and fed to the classifier. Also, the NN-GP hybrids are compared with MLFF, PNN and GP in their stand-alone mode without feature selection. The dataset analyzed here is collected from Wharton Research Data Services (WRDS). It consists of 240 dotcom companies of which 120 are failed and 120 are healthy. Ten-fold cross-validation is performed throughout the study. Results in terms of average accuracy, average sensitivity, average specificity and area under the receiver operating characteristic curve (AUC) indicate that the GP outperformed all the techniques with or without feature selection. The superiority of GP-GP is demonstrated by t-test at 10% level of significance. Furthermore, the results are much better than those reported in previous studies on the same dataset.
---
paper_title: An empirical study of classification algorithm evaluation for financial risk prediction

paper_content:

A wide range of classification methods have been used for the early detection of financial risks in recent years. How to select an adequate classifier (or set of classifiers) for a given dataset is an important task in financial risk prediction. Previous studies indicate that classifiers' performances in financial risk prediction may vary using different performance measures and under different circumstances. The main goal of this paper is to develop a two-step approach to evaluate classification algorithms for financial risk prediction. It constructs a performance score to measure the performance of classification algorithms and introduces three multiple criteria decision making (MCDM) methods (i.e., TOPSIS, PROMETHEE, and VIKOR) to provide a final ranking of classifiers. An empirical study is designed to assess various classification algorithms over seven real-life credit risk and fraud risk datasets from six countries. The results show that linear logistic, Bayesian Network, and ensemble methods are ranked as the top-three classifiers by TOPSIS, PROMETHEE, and VIKOR. In addition, this work discusses the construction of a knowledge-rich financial risk management process to increase the usefulness of classification results in financial risk detection.
---
paper_title: Soft computing system for bank performance prediction

paper_content:

This paper presents a soft computing based bank performance prediction system. It is an ensemble system whose constituent models are a multi-layered feed forward neural network trained with backpropagation (MLFF-BP), a probabilistic neural network (PNN) and a radial basis function neural network (RBFN), support vector machine (SVM), classification and regression trees (CART) and a fuzzy rule based classifier. Further, principal component analysis (PCA) based hybrid neural networks, viz. PCA-MLFF-BP, PCA-PNN and PCA-RBF are also included as constituents of the ensemble. Moreover, GRNN and PNN were trained with a genetic algorithm to optimize the smoothing factors. Two ensembles (i) simple majority voting based and (ii) weightage based are implemented. This system predicts the performance of a bank in the coming financial year based on its previous 2-years' financial data. Ten-fold cross-validation is performed in the training sessions and results are validated with an independent production set. It is demonstrated that the ensemble is able to yield lower Type I and Type II errors compared to its constituent models. Further, the ensemble also outperformed an earlier study [P.G. Swicegood, Predicting poor bank profitability: a comparison of neural network, discriminant analysis and professional human judgement, Ph.D. Thesis, Department of Finance, Florida State University, 1998] that used multivariate discriminant analysis (MDA), MLFF-BP and human judgment.
---
paper_title: Effects of feature construction on classification performance: An empirical study in bank failure prediction

paper_content:

While extensive research in data mining has been devoted to developing better classification algorithms, relatively little research has been conducted to examine the effects of feature construction, guided by domain knowledge, on classification performance. However, in many application domains, domain knowledge can be used to construct higher-level features to potentially improve performance. For example, past research and regulatory practice in early warning of bank failures has resulted in various explanatory variables, in the form of financial ratios, that are constructed based on bank accounting variables and are believed to be more effective than the original variables in identifying potential problem banks. In this study, we empirically compare the performance of two sets of classifiers for bank failure prediction, one built using raw accounting variables and the other built using constructed financial ratios. Four popular data mining methods are used to learn the classifiers: logistic regression, decision tree, neural network, and k-nearest neighbor. We evaluate the classifiers on the basis of expected misclassification cost under a wide range of possible settings. The results of the study strongly indicate that feature construction, guided by domain knowledge, significantly improves classifier performance and that the degree of improvement varies significantly across the methods.
---
paper_title: Combining cluster analysis with classifier ensembles to predict financial distress

paper_content:

The ability to accurately predict business failure is a very important issue in financial decision-making. Incorrect decision-making in financial institutions is very likely to cause financial crises and distress. Bankruptcy prediction and credit scoring are two important problems facing financial decision support. As many related studies develop financial distress models by some machine learning techniques, more advanced machine learning techniques, such as classifier ensembles and hybrid classifiers, have not been fully assessed. The aim of this paper is to develop a novel hybrid financial distress model based on combining the clustering technique and classifier ensembles. In addition, single baseline classifiers, hybrid classifiers, and classifier ensembles are developed for comparisons. In particular, two clustering techniques, Self-Organizing Maps (SOMs) and k-means and three classification techniques, logistic regression, multilayer-perceptron (MLP) neural network, and decision trees, are used to develop these four different types of bankruptcy prediction models. As a result, 21 different models are compared in terms of average prediction accuracy and Type I & II errors. By using five related datasets, combining Self-Organizing Maps (SOMs) with MLP classifier ensembles performs the best, which provides higher predication accuracy and lower Type I & II errors.
---
paper_title: A data mining framework for detecting subscription fraud in telecommunication

paper_content:

Service providing companies including telecommunication companies often receive substantial damage from customers' fraudulent behaviors. One of the common types of fraud is subscription fraud in which usage type is in contradiction with subscription type. This study aimed at identifying customers' subscription fraud by employing data mining techniques and adopting knowledge discovery process. To this end, a hybrid approach consisting of preprocessing, clustering, and classification phases was applied, and appropriate tools were employed commensurate to each phase. Specifically, in the clustering phase SOM and K-means were combined, and in the classification phase decision tree (C4.5), neural networks, and support vector machines as single classifiers and bagging, boosting, stacking, majority and consensus voting as ensembles were examined. In addition to using clustering to identify outlier cases, it was also possible - by defining new features - to maintain the results of clustering phase for the classification phase. This, in turn, contributed to better classification results. A real dataset provided by Telecommunication Company of Tehran was applied to demonstrate the effectiveness of the proposed method. The efficient use of synergy among these techniques significantly increased prediction accuracy. The performance of all single and ensemble classifiers is evaluated based on various metrics and compared by statistical tests. The results showed that support vector machines among single classifiers and boosted trees among all classifiers have the best performance in terms of various metrics. The research findings show that the proposed model has a high accuracy, and the resulting outcomes are significant both theoretically and practically.
---
paper_title: Detecting fraud in online games of chance and lotteries

paper_content:

Fraud detection has been an important topic of research in the data mining community for the past two decades. Supervised, semi-supervised, and unsupervised approaches to fraud detection have been proposed for the telecommunications, credit, insurance and health-care industries. We describe a novel hybrid system for detecting fraud in the highly growing lotteries and online games of chance sector. While the objectives of fraudsters in this sector are not unique, money laundering and insider attack scenarios are much more prevalent in lotteries than in the previously studied sectors. The lack of labeled data for supervised classifier design, user anonymity, and the size of the data-sets are the other key factors differentiating the problem from previous studies, and are the key drivers behind the design and implementation decisions for the system described. The system employs online algorithms that optimally aggregate statistical information from raw data and applies a number of pre-specified checks against known fraud scenarios as well as novel clustering-based algorithms for outlier detection which are then fused together to produce alerts with high detection rates at acceptable false alarm levels.
---
paper_title: A prescription fraud detection model

paper_content:

Prescription fraud is a main problem that causes substantial monetary loss in health care systems. We aimed to develop a model for detecting cases of prescription fraud and test it on real world data from a large multi-center medical prescription database. Conventionally, prescription fraud detection is conducted on random samples by human experts. However, the samples might be misleading and manual detection is costly. We propose a novel distance based on data-mining approach for assessing the fraudulent risk of prescriptions regarding cross-features. Final tests have been conducted on adult cardiac surgery database. The results obtained from experiments reveal that the proposed model works considerably well with a true positive rate of 77.4% and a false positive rate of 6% for the fraudulent medical prescriptions. The proposed model has the potential advantages including on-line risk prediction for prescription fraud, off-line analysis of high-risk prescriptions by human experts, and self-learning ability by regular updates of the integrative data sets. We conclude that incorporating such a system in health authorities, social security agencies and insurance companies would improve efficiency of internal review to ensure compliance with the law, and radically decrease human-expert auditing costs.
---
paper_title: Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes: A Systematic Review

paper_content:

ContextDevelopers of health care software have attributed improvements in patient ::: care to these applications. As with any health care intervention, such claims ::: require confirmation in clinical trials.ObjectivesTo review controlled trials assessing the effects of computerized clinical ::: decision support systems (CDSSs) and to identify study characteristics predicting ::: benefit.Data SourcesWe updated our earlier reviews by searching the MEDLINE, EMBASE, Cochrane ::: Library, Inspec, and ISI databases and consulting reference lists through ::: September 2004. Authors of 64 primary studies confirmed data or provided additional ::: information.Study SelectionWe included randomized and nonrandomized controlled trials that evaluated ::: the effect of a CDSS compared with care provided without a CDSS on practitioner ::: performance or patient outcomes.Data ExtractionTeams of 2 reviewers independently abstracted data on methods, setting, ::: CDSS and patient characteristics, and outcomes.Data SynthesisOne hundred studies met our inclusion criteria. The number and methodologic ::: quality of studies improved over time. The CDSS improved practitioner performance ::: in 62 (64%) of the 97 studies assessing this outcome, including 4 (40%) of ::: 10 diagnostic systems, 16 (76%) of 21 reminder systems, 23 (62%) of 37 disease ::: management systems, and 19 (66%) of 29 drug-dosing or prescribing systems. ::: Fifty-two trials assessed 1 or more patient outcomes, of which 7 trials (13%) ::: reported improvements. Improved practitioner performance was associated with ::: CDSSs that automatically prompted users compared with requiring users to activate ::: the system (success in 73% of trials vs 47%; P = .02) ::: and studies in which the authors also developed the CDSS software compared ::: with studies in which the authors were not the developers (74% success vs ::: 28%; respectively, P = .001).ConclusionsMany CDSSs improve practitioner performance. To date, the effects on ::: patient outcomes remain understudied and, when studied, inconsistent.
---
paper_title: AptaCDSS-E: A classifier ensemble-based clinical decision support system for cardiovascular disease level prediction

paper_content:

Conventional clinical decision support systems are generally based on a single classifier or a simple combination of these models, showing moderate performance. In this paper, we propose a classifier ensemble-based method for supporting the diagnosis of cardiovascular disease (CVD) based on aptamer chips. This AptaCDSS-E system overcomes conventional performance limitations by utilizing ensembles of different classifiers. Recent surveys show that CVD is one of the leading causes of death and that significant life savings can be achieved if precise diagnosis can be made. For CVD diagnosis, our system combines a set of four different classifiers with ensembles. Support vector machines and neural networks are adopted as base classifiers. Decision trees and Bayesian networks are also adopted to augment the system. Four aptamer-based biochip data sets including CVD data containing 66 samples were used to train and test the system. Three other supplementary data sets are used to alleviate data insufficiency. We investigated the effectiveness of the ensemble-based system with several different aggregation approaches by comparing the results with single classifier-based models. The prediction performance of the AptaCDSS-E system was assessed with a cross-validation test. The experimental results show that our system achieves high diagnosis accuracy (>94%) and comparably small prediction difference intervals (<6%), proving its usefulness in the clinical decision process of disease diagnosis. Additionally, 10 possible biomarkers are found for further investigation.
---
paper_title: Diagnosis of valvular heart disease through neural networks ensembles

paper_content:

In the last decades, several tools and various methodologies have been proposed by the researchers for developing effective medical decision support systems. Moreover, new methodologies and new tools are continued to develop and represent day by day. Diagnosing of the valvular heart disease is one of the important issue and many researchers investigated to develop intelligent medical decision support systems to improve the ability of the physicians. In this paper, we introduce a methodology which uses SAS Base Software 9.1.3 for diagnosing of the valvular heart disease. A neural networks ensemble method is in the centre of the proposed system. The ensemble-based methods creates new models by combining the posterior probabilities or the predicted values from multiple predecessor models. So, more effective models can be created. We performed experiments with proposed tool. We obtained 97.4% classification accuracy from the experiments made on data set containing 215 samples. We also obtained 100% and 96% sensitivity and specificity values, respectively, in valvular heart disease diagnosis.
---
paper_title: Effective diagnosis of heart disease through neural networks ensembles

paper_content:

In the last decades, several tools and various methodologies have been proposed by the researchers for developing effective medical decision support systems. Moreover, new methodologies and new tools are continued to develop and represent day by day. Diagnosing of the heart disease is one of the important issue and many researchers investigated to develop intelligent medical decision support systems to improve the ability of the physicians. In this paper, we introduce a methodology which uses SAS base software 9.1.3 for diagnosing of the heart disease. A neural networks ensemble method is in the centre of the proposed system. This ensemble based methods creates new models by combining the posterior probabilities or the predicted values from multiple predecessor models. So, more effective models can be created. We performed experiments with the proposed tool. We obtained 89.01% classification accuracy from the experiments made on the data taken from Cleveland heart disease database. We also obtained 80.95% and 95.91% sensitivity and specificity values, respectively, in heart disease diagnosis.
---
paper_title: Improving the Accuracy of an Artificial Neural Network Using Multiple Differently Trained Networks

paper_content:

When either detection rate (sensitivity) or false alarm rate (specificity) is optimized in an artificial neural network trained to identify myocardial infarction, the increase in the accuracy of one is always done at the expense of the accuracy of the other. To overcome this loss, two networks that were separately trained on populations of patients with different likelihoods of myocardial infarction were used in concert. One network was trained on clinical pattern sets derived from patients who had a low likelihood of myocardial infarction, while the other was trained on pattern sets derived from patients with a high likelihood of myocardial infarction. Unknown patterns were analyzed by both networks. If the output generated by the network trained on the low risk patients was below an empirically set threshold, this output was chosen as the diagnostic output. If the output was above that threshold, the output of the network trained on the high risk patients was used as the diagnostic output. The dual network correctly identified 39 of the 40 patients who had sustained a myocardial infarction and 301 of 306 patients who did not have a myocardial infarction for a detection rate (sensitivity) and false alarm rate (1-specificity) of 97.50 and 1.63%, respectively. A parallel control experiment using a single network but identical training information correctly identified 39 of 40 patients who had sustained a myocardial infarction and 287 of 306 patients who had not sustained a myocardial infarction (p = 0.003).
---
paper_title: Letters: Ensemble of classifiers for protein fold recognition

paper_content:

Predicting the three-dimensional structure of a protein from its amino-acid sequence is an important problem in bioinformatics and a challenging task for machine learning algorithms. In this paper, we investigate an ensemble of classifiers, applied for protein fold recognition, and tested it on a real-world dataset. The obtained results are very encouraging, our results improved the average predictive accuracy obtained in previously published work.
---
paper_title: Hybrid system for protein secondary structure prediction

paper_content:

We have developed a hybrid system to predict the secondary structures (α-helix, β-sheet and coil) of proteins and achieved 66.4% accuracy, with correlation coefficients of Ccoil = 0.429, Cα = 0.470 and Cβ = 0.387. This system contains three subsystems (“experts”): a neural network module, a statistical module and a memory-based reasoning module. First, the three experts independently learn the mapping between amino acid sequences and secondary structures from the known protein structures, then a Combiner learns to combine automatically the outputs of the experts to make final predictions. The hybrid system was tested with 107 protein structures through k-way cross-validation. Its performance was better than each expert and all previously reported methods with greater than 0.99 statistical significance. It was observed that for 20% of the residues, all three experts produced the same but wrong predictions. This may suggest an upper bound on the accuracy of secondary structure predictions based on local information from the currently available protein structures, and indicate places where non-local interactions may play a dominant role in conformation. For 64% of the residues, at least two experts were the same and correct, which shows that the Combiner performed better than majority vote. For 77 % of the residues, at least one expert was correct, thus there may still be room for improvement in this hybrid approach. Rigorous evaluation procedures were used in testing the hybrid system, and statistical significance measures were developed in analyzing the differences among different methods. When measured in terms of the number of secondary structures (rather than the number of residues) that were predicted correctly, the prediction produced by the hybrid system was also better than those of individual experts.
---
paper_title: Hybrid dendritic computing with kernel-LICA applied to Alzheimer's disease detection in MRI

paper_content:

Dendritic computing has been proved to produce perfect approximation of any data distribution. This result guarantees perfect accuracy training. However, we have found great performance degradation when tested on conventional k-fold cross-validation schemes. In this paper we propose to use Lattice Independent Component Analysis (LICA) and the Kernel transformation of the data as an appropriate feature extraction that improves the generalization of dendritic computing classifiers. We obtain a big increase in classification performance applying with this schema over a database of features extracted from Magnetic Resonance Imaging (MRI) including Alzheimer's disease (AD) patients and control subjects.
---
paper_title: Decoding visual brain states from fMRI using an ensemble of classifiers

paper_content:

Decoding perceptual or cognitive states based on brain activity measured using functional magnetic resonance imaging (fMRI) can be achieved using machine learning algorithms to train classifiers of specific stimuli. However, the high dimensionality and intrinsically low signal to noise ratio (SNR) of fMRI data poses great challenges to such techniques. The problem is aggravated in the case of multiple subject experiments because of the high inter-subject variability in brain function. To address these difficulties, the majority of current approaches uses a single classifier. Since, in many cases, different stimuli activate different brain areas, it makes sense to use a set of classifiers each specialized in a different stimulus. Therefore, we propose in this paper using an ensemble of classifiers for decoding fMRI data. Each classifier in the ensemble has a favorite class or stimulus and uses an optimized feature set for that particular stimulus. The output for each individual stimulus is therefore obtained from the corresponding classifier and the final classification is achieved by simply selecting the best score. The method was applied to three empirical fMRI datasets from multiple subjects performing visual tasks with four classes of stimuli. Ensembles of GNB and k-NN base classifiers were tested. The ensemble of classifiers systematically outperformed a single classifier for the two most challenging datasets. In the remaining dataset, a ceiling effect was observed which probably precluded a clear distinction between the two classification approaches. Our results may be explained by the fact that different visual stimuli elicit specific patterns of brain activation and indicate that an ensemble of classifiers provides an advantageous alternative to commonly used single classifiers, particularly when decoding stimuli associated with specific brain areas.
---
paper_title: Neurocognitive disorder detection based on feature vectors extracted from VBM analysis of structural MRI

paper_content:

Dementia is a growing concern due to the aging process of the western societies. Non-invasive detection is therefore a high priority research endeavor. In this paper we report results of classification systems applied to the feature vectors obtained by a feature extraction method computed on structural magnetic resonance imaging (sMRI) volumes for the detection of two neurological disorders with cognitive impairment: myotonic dystrophy of type 1 (MD1) and Alzheimer disease (AD). The feature extraction process is based on the voxel clusters detected by voxel-based morphometry (VBM) analysis of sMRI upon a set of patient and control subjects. This feature extraction process is specific for each kind of disease and is grounded on the findings obtained by medical experts. The 10-fold cross-validation results of several statistical and neural network based classification algorithms trained and tested on these features show high specificity and moderate sensitivity of the classifiers, suggesting that the approach is better suited for rejecting than for detecting early stages of the diseases.
---
paper_title: Classifier Ensembles for fMRI Data Analysis: An Experiment

paper_content:

Abstract Functional magnetic resonance imaging (fMRI) is becoming a forefront brain–computer interface tool. To decipher brain patterns, fast, accurate and reliable classifier methods are needed. The support vector machine (SVM) classifier has been traditionally used. Here we argue that state-of-the-art methods from pattern recognition and machine learning, such as classifier ensembles, offer more accurate classification. This study compares 18 classification methods on a publicly available real data set due to Haxby et al. [ Science 293 (2001) 2425–2430]. The data comes from a single-subject experiment, organized in 10 runs where eight classes of stimuli were presented in each run. The comparisons were carried out on voxel subsets of different sizes, selected through seven popular voxel selection methods. We found that, while SVM was robust, accurate and scalable, some classifier ensemble methods demonstrated significantly better performance. The best classifiers were found to be the random subspace ensemble of SVM classifiers, rotation forest and ensembles with random linear and random spherical oracle.
---
paper_title: Naive random subspace ensemble with linear classifiers for real-time classification of fMRI data

paper_content:

Functional magnetic resonance imaging (fMRI) provides a spatially accurate measure of brain activity. Real-time classification allows the use of fMRI in neurofeedback experiments. With limited labelled data available, a fixed pre-trained classifier may be inaccurate. We propose that streaming fMRI data may be classified using a classifier ensemble which is updated through naive labelling. Naive labelling is a protocol where in the absence of ground truth, updates are carried out using the label assigned by the classifier. We perform experiments on three fMRI datasets to demonstrate that naive labelling is able to improve upon a pre-trained initial classifier.
---
paper_title: A Two Stage Sequential Ensemble Applied to the Classification of Alzheimer’s Disease Based on MRI Features

paper_content:

We present a two stage sequential ensemble where data samples whose output from the first classifier fall in a low confidence output interval (LCOI) are processed by a second stage classifier. Training is composed of three processes: training the first classifier, determining the LCOI of the first classifier, and training the second classifier upon the data items whose output fall in the LCOI. The LCOI is determined varying a threshold on the false positive rate (FPR) and false negative rate (FNR) curves. We have tested the approach on a database of feature vectors for the classification of Alzheimer's disease (AD) and control subjects extracted from structural magnetic resonance imaging (sMRI) data. In this paper, we focus on the combinations obtained when the first classifier is a relevance vector machine (RVM). Obtained results improve over previous results for this database.
---
paper_title: User Modeling for Adaptive News Access

paper_content:

We present a framework for adaptive news access, based on machine learning techniques specifically designed for this task. First, we focus on the system's general functionality and system architecture. We then describe the interface and design of two deployed news agents that are part of the described architecture. While the first agent provides personalized news through a web-based interface, the second system is geared towards wireless information devices such as PDAs (personal digital assistants) and cell phones. Based on implicit and explicit user feedback, our agents use a machine learning algorithm to induce individual user models. Motivated by general shortcomings of other user modeling systems for Information Retrieval applications, as well as the specific requirements of news classification, we propose the induction of hybrid user models that consist of separate models for short-term and long-term interests. Furthermore, we illustrate how the described algorithm can be used to address an important issue that has thus far received little attention in the Information Retrieval community: a user's information need changes as a direct result of interaction with information. We empirically evaluate the system's performance based on data collected from regular system users. The goal of the evaluation is not only to understand the performance contributions of the algorithm's individual components, but also to assess the overall utility of the proposed user modeling techniques from a user perspective. Our results provide empirical evidence for the utility of the hybrid user model, and suggest that effective personalization can be achieved without requiring any extra effort from the user.
---
paper_title: Combining predictions for accurate recommender systems

paper_content:

We analyze the application of ensemble learning to recommender systems on the Netflix Prize dataset. For our analysis we use a set of diverse state-of-the-art collaborative filtering (CF) algorithms, which include: SVD, Neighborhood Based Approaches, Restricted Boltzmann Machine, Asymmetric Factor Model and Global Effects. We show that linearly combining (blending) a set of CF algorithms increases the accuracy and outperforms any single CF algorithm. Furthermore, we show how to use ensemble methods for blending predictors in order to outperform a single blending algorithm. The dataset and the source code for the ensemble blending are available online.
---
paper_title: A Framework for Collaborative, Content-Based and Demographic Filtering

paper_content:

We discuss learning a profile of user interests for recommending information sources such as Web pages or news articles. We describe the types of information available to determine whether to recommend a particular page to a particular user. This information includes the content of the page, the ratings of the user on other pages and the contents of these pages, the ratings given to that page by other users and the ratings of these other users on other pages and demographic information about users. We describe how each type of information may be used individually and then discuss an approach to combining recommendations from multiple sources. We illustrate each approach and the combined approach in the context of recommending restaurants.
---
paper_title: Hybrid Recommender Systems: Survey and Experiments

paper_content:

Recommender systems represent user preferences for the purpose of suggesting items to purchase or examine. They have become fundamental applications in electronic commerce and information access, providing suggestions that effectively prune large information spaces so that users are directed toward those items that best meet their needs and preferences. A variety of techniques have been proposed for performing recommendation, including content-based, collaborative, knowledge-based and other techniques. To improve performance, these methods have sometimes been combined in hybrid recommenders. This paper surveys the landscape of actual and possible hybrid recommenders, and introduces a novel hybrid, EntreeC, a system that combines knowledge-based recommendation and collaborative filtering to recommend restaurants. Further, we show that semantic ratings obtained from the knowledge-based part of the system enhance the effectiveness of collaborative filtering.
---
paper_title: Combination of Web page recommender systems

paper_content:

With the rapid growth of the World Wide Web (www), finding useful information from the Internet has become a critical issue. Web recommender systems help users make decisions in this complex information space where the volume of information available to them is huge. Recently, a number of Web page recommender systems have been developed to extract the user behavior from the user's navigational path and predict the next request as s/he visits Web pages. However, each of these systems has its own merits and limitations. In this paper, we investigate a hybrid recommender system, which combines the results of several recommender techniques based on Web usage mining. We conduct a detailed comparative evaluation of how different combined methods and different recommendation techniques affect the prediction accuracy of the hybrid recommender. We then discuss the results in terms of using a hybrid recommender system instead of a single recommender model. Our results suggest that the hybrid recommender system is better in predicting the next request of a Web user.
---
paper_title: Fab: Content-based, collaborative recommendation

paper_content:

A passive, integrating electromagnetic radiation power dosimeter. A radiofrequency or microwave antenna is combined with a diode detector/rectifier, a squaring circuit, and a electrochemical storage cell to provide an apparatus for determining the average energy of electromagnetic radiation incident on a surface. After a particular period of irradiation, the dosimeter can be interrogated electrically or visually, depending on the type of electrochemical cell employed, to yield the desired information. The apparatus has a substantially linear response to the electromagnetic power density over a wide range of electromagnetic field, and all of the energy required to record the incident energy is supplied by the electromagnetic field.
---
paper_title: Incorporating contextual information in recommender systems using a multidimensional approach

paper_content:

The article presents a multidimensional (MD) approach to recommender systems that can provide recommendations based on additional contextual information besides the typical information on users and items used in most of the current recommender systems. This approach supports multiple dimensions, profiling information, and hierarchical aggregation of recommendations. The article also presents a multidimensional rating estimation method capable of selecting two-dimensional segments of ratings pertinent to the recommendation context and applying standard collaborative filtering or other traditional two-dimensional rating estimation techniques to these segments. A comparison of the multidimensional and two-dimensional rating estimation approaches is made, and the tradeoffs between the two are studied. Moreover, the article introduces a combined rating estimation method, which identifies the situations where the MD approach outperforms the standard two-dimensional approach and uses the MD approach in those situations and the standard two-dimensional approach elsewhere. Finally, the article presents a pilot empirical study of the combined approach, using a multidimensional movie recommender system that was developed for implementing this approach and testing its performance.
---
paper_title: Optimisation of combined collaborative recommender systems

paper_content:

Abstract A new approach to collaborative user modelling is presented in this paper. We have developed a framework that can be used for easy testing of different concepts. We have also introduced three different areas where collaborative modelling can be further improved. For the first phase of testing, we have created a hybrid system based on three different collaborative recommender techniques. Since this system implements multiple collaboration techniques, we decided to call this approach Combined Collaborative Recommender. Although each prediction technique can produce adequate results, we have proved that the combination of these techniques into a unified system provides a much more stable system. It should also be pointed out that these analyses were done using a very large dataset (more than 2 million ratings) providing reliable results. Results of these optimisations are presented along with pointers for further development.
---
paper_title: Hybrid Recommender Systems for Electronic Commerce

paper_content:

In electronic commerce applications, prospective buyers may be interested in receiving recommendations to assist with their purchasing decisions. Previous research has described two main models for automated recommender systems - collaborative filtering and the knowledge-based approach. In this paper, we present an architecture for designing a hybrid recommender system that combines these two approaches. We then discuss how such a recommender system can switch between the two methods, depending on the current support for providing good recommendations from the behaviour of other users, required for the collaborative filtering option. We also comment on how the overall design is useful to support recommendations for a variety of product areas and present some directions for future work. Overview One of the tasks in the application area of knowledgebased electronic markets is that of providing recommendations to potential shoppers. In an environment where there is a wide choice for the prospective buyers, an automated system which serves to present a more narrow selection for the buyer would be desirable. Recommender systems are systems which provide recommendations to potential buyers. Two widely used techniques for building recommender systems to date are collaborative filtering and knowledge-based approaches. Collaborative filtering is a real-time personalization technique that leverages similarities between people to make recommendations (Greening 1998). other words, a collaborative filtering recommender system assumes that human preferences are correlated; thus, it predicts preferences and makes recommendations to one user based on the preferences of a group of users. In contrast, a knowledge-based recommender system exploits its knowledge base of the product domain to generate recommendations to a user, by reasoning about what products meet the user’s requirements. In this paper, we analyze the advantages and shortcomings of both techniques and present an architecture for a hybrid recommender system that integrates the two approaches. Such a system will inherit all the strengths from a collaborative filtering recommender system, but will be able to avoid its weaknesses. Although there have been some proposals for designing systems which make use of both the knowledge-base approach and collaborative filtering (Burke 1999), collaborative filtering is used more in a post-processing stage, so that the knowledge-based design predominates. In this paper, we outline some specifications for changing between the collaborative filtering and the knowledge-based styles of recommendation, within a single system. This design strategy will be useful for electronic commerce applications where the number of buyers and the make-up of the community of buyers dictates whether collaborative filtering will be effective or not. Background In this section, we introduce the collaborative filtering and knowledge-based approaches in building recommender systems. We discuss the strengths and weaknesses of each approach as the motivation for the design of a hybrid architecture that combines the two approaches.
---
paper_title: A hybrid recommender system for the selective dissemination of research resources in a Technology Transfer Office

paper_content:

Recommender systems could be used to help users in their access processes to relevant information. Hybrid recommender systems represent a promising solution for multiple applications. In this paper we propose a hybrid fuzzy linguistic recommender system to help the Technology Transfer Office staff in the dissemination of research resources interesting for the users. The system recommends users both specialized and complementary research resources and additionally, it discovers potential collaboration possibilities in order to form multidisciplinary working groups. Thus, this system becomes an application that can be used to help the Technology Transfer Office staff to selectively disseminate the research knowledge and to increase its information discovering properties and personalization capacities in an academic environment.
---

Title: A Survey of Multiple Classifier Systems as Hybrid Systems

Section 1: Introduction
Description 1: This section introduces the topic of Hybrid Intelligent Systems and provides the context and motivation for the survey.

Section 2: Advantages
Description 2: This section outlines the benefits of Multiple Classifier Systems (MCS) and their advantages in various scenarios.

Section 3: System Structure
Description 3: This section explains the general structure of Multiple Classifier Systems (MCS), including system topology, ensemble design, fuser design, and diversity measures.

Section 4: Ensemble Design
Description 4: This section describes the design process for classifier ensembles, emphasizing the importance of diversity and various techniques to ensure it.

Section 5: Diversity Measures
Description 5: This section discusses metrics and methods to measure and ensure diversity within classifier ensembles.

Section 6: Ensuring Diversity
Description 6: This section elaborates on various strategies for diversifying inputs, outputs, and models to enhance the effectiveness of the classifier pool.

Section 7: Fuser Design
Description 7: This section covers different strategies for designing the fuser, which combines the outputs of individual classifiers to make a final decision.

Section 8: Trainable Fuser
Description 8: This section talks about fuser models that can be trained, seeking to improve classifier combination through specific learning processes.

Section 9: Concept Drift
Description 9: This section explains the concept drift phenomenon and how Multiple Classifier Systems handle non-stationary processes and dynamic environments.

Section 10: Applications
Description 10: This section reviews recent applications of Multiple Classifier Systems in various fields such as remote sensing, computer security, financial risk assessment, fraud detection, and medicine.

Section 11: Final Remarks
Description 11: This section provides a summary and closing thoughts on the future directions and potential hybridization strategies for Multiple Classifier Systems.