A Review of Feature Selection and Its Methods
---
paper_title: Feature Selection: Evaluation, Application, and Small Sample Performance

paper_content:

A large number of algorithms have been proposed for feature subset selection. Our experimental results show that the sequential forward floating selection algorithm, proposed by Pudil et al. (1994), dominates the other algorithms tested. We study the problem of choosing an optimal feature set for land use classification based on SAR satellite images using four different texture models. Pooling features derived from different texture models, followed by a feature selection results in a substantial improvement in the classification accuracy. We also illustrate the dangers of using feature selection in small sample size situations.
---
paper_title: Efficient Feature Selection via Analysis of Relevance and Redundancy

paper_content:

Feature selection is applied to reduce the number of features in many applications where data has hundreds or thousands of features. Existing feature selection methods mainly focus on finding relevant features. In this paper, we show that feature relevance alone is insufficient for efficient feature selection of high-dimensional data. We define feature redundancy and propose to perform explicit redundancy analysis in feature selection. A new framework is introduced that decouples relevance analysis and redundancy analysis. We develop a correlation-based method for relevance and redundancy analysis, and conduct an empirical study of its efficiency and effectiveness comparing with representative methods.
---
paper_title: Feature selection and analysis on correlated gas sensor data with recursive feature elimination

paper_content:

Abstract Support vector machine recursive feature elimination (SVM-RFE) is a powerful feature selection algorithm. However, when the candidate feature set contains highly correlated features, the ranking criterion of SVM-RFE will be biased, which would hinder the application of SVM-RFE on gas sensor data. In this paper, the linear and nonlinear SVM-RFE algorithms are studied. After investigating the correlation bias, an improved algorithm SVM-RFE + CBR is proposed by incorporating the correlation bias reduction (CBR) strategy into the feature elimination procedure. Experiments are conducted on a synthetic dataset and two breath analysis datasets, one of which contains temperature modulated sensors. Large and comprehensive sets of transient features are extracted from the sensor responses. The classification accuracy with feature selection proves the efficacy of the proposed SVM-RFE + CBR. It outperforms the original SVM-RFE and other typical algorithms. An ensemble method is further studied to improve the stability of the proposed method. By statistically analyzing the features’ rankings, some knowledge is obtained, which can guide future design of e-noses and feature extraction algorithms.
---
paper_title: Feature subset selection in large dimensionality domains

paper_content:

Searching for an optimal feature subset from a high dimensional feature space is known to be an NP-complete problem. We present a hybrid algorithm, SAGA, for this task. SAGA combines the ability to avoid being trapped in a local minimum of simulated annealing with the very high rate of convergence of the crossover operator of genetic algorithms, the strong local search ability of greedy algorithms and the high computational efficiency of generalized regression neural networks. We compare the performance over time of SAGA and well-known algorithms on synthetic and real datasets. The results show that SAGA outperforms existing algorithms.
---
paper_title: SlimPLS: A Method for Feature Selection in Gene Expression-Based Disease Classification

paper_content:

A major challenge in biomedical studies in recent years has been the classification of gene expression profiles into categories, such as cases and controls. This is done by first training a classifier by using a labeled training set containing labeled samples from the two populations, and then using that classifier to predict the labels of new samples. Such predictions have recently been shown to improve the diagnosis and treatment selection practices for several diseases. This procedure is complicated, however, by the high dimensionality if the data. While microarrays can measure the levels of thousands of genes per sample, case-control microarray studies usually involve no more than several dozen samples. Standard classifiers do not work well in these situations where the number of features (gene expression levels measured in these microarrays) far exceeds the number of samples. Selecting only the features that are most relevant for discriminating between the two categories can help construct better classifiers, in terms of both accuracy and efficiency. In this work we developed a novel method for multivariate feature selection based on the Partial Least Squares algorithm. We compared the method's variants with common feature selection techniques across a large number of real case-control datasets, using several classifiers. We demonstrate the advantages of the method and the preferable combinations of classifier and feature selection technique.
---
paper_title: Floating search methods in feature selection

paper_content:

Sequential search methods characterized by a dynamically changing number of features included or eliminated at each step, henceforth "floating" methods, are presented. They are shown to give very good results and to be computationally more effective than the branch and bound method.
---
paper_title: Irrelevant Features and the Subset Selection Problem

paper_content:

We address the problem of finding a subset of features that allows a supervised induction algorithm to induce small high-accuracy concepts. We examine notions of relevance and irrelevance, and show that the definitions used in the machine learning literature do not adequately partition the features into useful categories of relevance. We present definitions for irrelevance and for two degrees of relevance. These definitions improve our understanding of the behavior of previous subset selection algorithms, and help define the subset of features that should be sought. The features selected should depend not only on the features and the target concept, but also on the induction algorithm. We describe a method for feature subset selection using cross-validation that is applicable to any induction algorithm, and discuss experiments conducted with ID3 and C4.5 on artificial and real datasets.
---
paper_title: Feature subset selection in large dimensionality domains

paper_content:

Searching for an optimal feature subset from a high dimensional feature space is known to be an NP-complete problem. We present a hybrid algorithm, SAGA, for this task. SAGA combines the ability to avoid being trapped in a local minimum of simulated annealing with the very high rate of convergence of the crossover operator of genetic algorithms, the strong local search ability of greedy algorithms and the high computational efficiency of generalized regression neural networks. We compare the performance over time of SAGA and well-known algorithms on synthetic and real datasets. The results show that SAGA outperforms existing algorithms.
---
paper_title: Normalized Mutual Information Feature Selection

paper_content:

A filter method of feature selection based on mutual information, called normalized mutual information feature selection (NMIFS), is presented. NMIFS is an enhancement over Battiti's MIFS, MIFS-U, and mRMR methods. The average normalized mutual information is proposed as a measure of redundancy among features. NMIFS outperformed MIFS, MIFS-U, and mRMR on several artificial and benchmark data sets without requiring a user-defined parameter. In addition, NMIFS is combined with a genetic algorithm to form a hybrid filter/wrapper method called GAMIFS. This includes an initialization procedure and a mutation operator based on NMIFS to speed up the convergence of the genetic algorithm. GAMIFS overcomes the limitations of incremental search algorithms that are unable to find dependencies between groups of features.
---
paper_title: Input feature selection by mutual information based on Parzen window

paper_content:

Mutual information is a good indicator of relevance between variables, and have been used as a measure in several feature selection algorithms. However, calculating the mutual information is difficult, and the performance of a feature selection algorithm depends on the accuracy of the mutual information. In this paper, we propose a new method of calculating mutual information between input and class variables based on the Parzen window, and we apply this to a feature selection algorithm for classification problems.
---
paper_title: Gene Selection Using Wilcoxon Rank Sum Test and Support Vector Machine for Cancer Classification

paper_content:

Gene selection is an important problem in microarray data processing. A new gene selection method based on Wilcoxon rank sum test and Support Vector Machine (SVM) is proposed in this paper. First, Wilcoxon rank sum test is used to select a subset. Then each selected gene is trained and tested using SVM classifier with linear kernel separately, and genes with high testing accuracy rates are chosen to form the final reduced gene subset. Leave-one-out cross validation (LOOCV) classification results on two datasets: Breast Cancer and ALL/AML leukemia, demonstrate the proposed method can get 100% success rate with the final reduced subset. The selected genes are listed and their expression levels are sketched, which show that the selected genes can make clear separation between two classes.
---
paper_title: An Improved Maximum Relevance and Minimum Redundancy Feature Selection Algorithm Based on Normalized Mutual Information

paper_content:

We present in this paper a comprehensive analysis of the mutual information based feature selection algorithms. We point out the limitations of some recent work in this area then propose an improvement to overcome the weak points. The experiment results confirm that we achieve a better feature sets compared with the two recent developed algorithms, which are Maximum Relevance and Minimum Redundancy (mRMR) and Normalized Mutual Information Feature Selection (NMIFS), in terms of the classification accuracy.
---
paper_title: Machine Learning Techniques and Chi-Square Feature Selection for Cancer Classification Using SAGE Gene Expression Profiles

paper_content:

Recently developed Serial Analysis of Gene Expression (SAGE) technology enables us to simultaneously quantify the expression levels of tens of thousands of genes in a population of cells. SAGE is better than Microarray in that SAGE can monitor both known and unknown genes while Microarray can only measure known genes. SAGE gene expression profiling based cancer classification is a better choice since cancers may be due to some unknown genes. Whereas a wide range of methods has been applied to traditional Microarray based cancer classification, relatively few studies have been done on SAGE based cancer classification. In our study we evaluate popular machine learning methods (SVM, Naive Bayes, Nearest Neighbor, C4.5 and RIPPER) for classifying cancers based on SAGE data. In order to deal with the high dimensional problem, we propose to use Chi-square for tag/gene selection. Both binary classification and multicategory classification are investigated. The experiments are based on two human SAGE datasets: brain and breast. The results show that SVM and Naive Bayes are the top-performing SAGE classifiers and that Chi-square based gene selection can improve the performance of all the five classifiers investigated.
---
paper_title: Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy

paper_content:

Feature selection is an important problem for pattern classification systems. We study how to select good features according to the maximal statistical dependency criterion based on mutual information. Because of the difficulty in directly implementing the maximal dependency condition, we first derive an equivalent form, called minimal-redundancy-maximal-relevance criterion (mRMR), for first-order incremental feature selection. Then, we present a two-stage feature selection algorithm by combining mRMR and other more sophisticated feature selectors (e.g., wrappers). This allows us to select a compact set of superior features at very low cost. We perform extensive experimental comparison of our algorithm and other methods using three different classifiers (naive Bayes, support vector machine, and linear discriminate analysis) and four different data sets (handwritten digits, arrhythmia, NCI cancer cell lines, and lymphoma tissues). The results confirm that mRMR leads to promising improvement on feature selection and classification accuracy.
---
paper_title: Estimating Attributes: Analysis and Extensions of RELIEF

paper_content:

In the context of machine learning from examples this paper deals with the problem of estimating the quality of attributes with and without dependencies among them. Kira and Rendell (1992a,b) developed an algorithm called RELIEF, which was shown to be very efficient in estimating attributes. Original RELIEF can deal with discrete and continuous attributes and is limited to only two-class problems. In this paper RELIEF is analysed and extended to deal with noisy, incomplete, and multi-class data sets. The extensions are verified on various artificial and one well known real-world problem.
---
paper_title: Data Visualization and Feature Selection: New Algorithms for Nongaussian Data

paper_content:

Data visualization and feature selection methods are proposed based on the joint mutual information and ICA. The visualization methods can find many good 2-D projections for high dimensional data interpretation, which cannot be easily found by the other existing methods. The new variable selection method is found to be better in eliminating redundancy in the inputs than other methods based on simple mutual information. The efficacy of the methods is illustrated on a radar signal analysis problem to find 2-D viewing coordinates for data visualization and to select inputs for a neural network classifier.
---
paper_title: On the Use of Variable Complementarity for Feature Selection in Cancer Classification

paper_content:

The paper presents an original filter approach for effective feature selection in classification tasks with a very large number of input variables. The approach is based on the use of a new information theoretic selection criterion: the double input symmetrical relevance (DISR). The rationale of the criterion is that a set of variables can return an information on the output class that is higher than the sum of the informations of each variable taken individually. This property will be made explicit by defining the measure of variable complementarity. A feature selection filter based on the DISR criterion is compared in theoretical and experimental terms to recently proposed information theoretic criteria. Experimental results on a set of eleven microarray classification tasks show that the proposed technique is competitive with existing filter selection methods.
---
paper_title: Using mutual information for selecting features in supervised neural net learning

paper_content:

This paper investigates the application of the mutual information criterion to evaluate a set of candidate features and to select an informative subset to be used as input data for a neural network classifier. Because the mutual information measures arbitrary dependencies between random variables, it is suitable for assessing the "information content" of features in complex classification tasks, where methods bases on linear relations (like the correlation) are prone to mistakes. The fact that the mutual information is independent of the coordinates chosen permits a robust estimation. Nonetheless, the use of the mutual information for tasks characterized by high input dimensionality requires suitable approximations because of the prohibitive demands on computation and samples. An algorithm is proposed that is based on a "greedy" selection of the features and that takes both the mutual information with respect to the output class and with respect to the already-selected features into account. Finally the results of a series of experiments are discussed. >
---
paper_title: A Fast Clustering-Based Feature Subset Selection Algorithm for High-Dimensional Data

paper_content:

Feature selection involves identifying a subset of the most useful features that produces compatible results as the original entire set of features. A feature selection algorithm may be evaluated from both the efficiency and effectiveness points of view. While the efficiency concerns the time required to find a subset of features, the effectiveness is related to the quality of the subset of features. Based on these criteria, a fast clustering-based feature selection algorithm (FAST) is proposed and experimentally evaluated in this paper. The FAST algorithm works in two steps. In the first step, features are divided into clusters by using graph-theoretic clustering methods. In the second step, the most representative feature that is strongly related to target classes is selected from each cluster to form a subset of features. Features in different clusters are relatively independent, the clustering-based strategy of FAST has a high probability of producing a subset of useful and independent features. To ensure the efficiency of FAST, we adopt the efficient minimum-spanning tree (MST) clustering method. The efficiency and effectiveness of the FAST algorithm are evaluated through an empirical study. Extensive experiments are carried out to compare FAST and several representative feature selection algorithms, namely, FCBF, ReliefF, CFS, Consist, and FOCUS-SF, with respect to four types of well-known classifiers, namely, the probability-based Naive Bayes, the tree-based C4.5, the instance-based IB1, and the rule-based RIPPER before and after feature selection. The results, on 35 publicly available real-world high-dimensional image, microarray, and text data, demonstrate that the FAST not only produces smaller subsets of features but also improves the performances of the four types of classifiers.
---
paper_title: Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy

paper_content:

Feature selection is an important problem for pattern classification systems. We study how to select good features according to the maximal statistical dependency criterion based on mutual information. Because of the difficulty in directly implementing the maximal dependency condition, we first derive an equivalent form, called minimal-redundancy-maximal-relevance criterion (mRMR), for first-order incremental feature selection. Then, we present a two-stage feature selection algorithm by combining mRMR and other more sophisticated feature selectors (e.g., wrappers). This allows us to select a compact set of superior features at very low cost. We perform extensive experimental comparison of our algorithm and other methods using three different classifiers (naive Bayes, support vector machine, and linear discriminate analysis) and four different data sets (handwritten digits, arrhythmia, NCI cancer cell lines, and lymphoma tissues). The results confirm that mRMR leads to promising improvement on feature selection and classification accuracy.
---
paper_title: A GA-LR wrapper approach for feature selection in network intrusion detection

paper_content:

Abstract Intrusions constitute one of the main issues in computer network security. Through malicious actions, hackers can have unauthorised access that compromises the integrity, the confidentiality, and the availability of resources or services. Intrusion detection systems (IDSs) have been developed to monitor and filter network activities by identifying attacks and alerting network administrators. Different IDS approaches have emerged using data mining, machine learning, statistical analysis, and artificial intelligence techniques such as genetic algorithms, artificial neural networks, fuzzy logic, swarm intelligence, etc. Due to the high dimensionality of the exchanged data, applying those techniques will be extremely time consuming. Feature selection is needed to select the optimal subset of features that represents the entire dataset to increase the accuracy and the classification performance of the IDS. In this work, we apply a wrapper approach based on a genetic algorithm as a search strategy and logistic regression as a learning algorithm for network intrusion detection systems to select the best subset of features. The experiment will be conducted on the KDD99 dataset and the UNSW-NB15 dataset. Three different decision tree classifiers are used to measure the performance of the selected subsets of features. The obtained results are compared with other feature selection approaches to verify the efficiency of our proposed approach.
---
paper_title: Occupancy estimation from environmental parameters using wrapper and hybrid feature selection

paper_content:

Abstract Occupancy information is essential to facilitate demand-driven operations of air-conditioning and mechanical ventilation (ACMV) systems. Environmental sensors are increasingly being explored as cost effective and non-intrusive means to obtain the occupancy information. This requires the extraction and selection of useful features from the sensor data. In past works, feature selection has generally been implemented using filter-based approaches. In this work, we introduce the use of wrapper and hybrid feature selection for better occupancy estimation. To achieve a fast computation time, we introduce a ranking-based incremental search in our algorithms, which is more efficient than the exhaustive search used in past works. For wrapper feature selection, we propose the WRANK-ELM, which searches an ordered list of features using the extreme learning machine (ELM) classifier. For hybrid feature selection, we propose the RIG-ELM, which is a filter–wrapper hybrid that uses the relative information gain (RIG) criterion for feature ranking and the ELM for the incremental search. We present experimental results in an office space with a multi-sensory network to validate the proposed algorithms.
---
paper_title: A novel wrapper method for feature selection and its applications

paper_content:

This paper introduces a wrapper method, namely cosine similarity measure support vector machines (CSMSVM), to eliminate irrelevant or redundant features during classifier construction by introducing the cosine distance into support vector machines (SVM). Traditionally, feature selection approaches typically extract features and learn SVM parameters independently or in the attribute space, which might result in a loss of information related to classification process or lead to the increase of classification error when introduce the kernel SVM. The proposed CSMSVM framework, however, jointly performs feature selection, SVM parameter learning and remove low relevance features by optimizing the shape of an anisotropic RBF kernel in feature space. Moreover, the Bayesian interpretation of the novel methodology reveals its Bayesian character, which builds the proposed method on solid theory foundation, and the iteration algorithm, which is proposed to optimize the feature weight, has achieved to maximize the maximum a posterior (MAP). Comparing the novel method with well-known feature selection techniques with experiments, CSMSVM outperformed the other methodologies in improving the pattern recognition accuracy with fewer features. An excellent feature selection method combined with SVM has been proposed.An efficient optimization algorithm has been proposed for non-convex problem.Features have been reduced without affecting the performance of SVM.Bayesian interpretation of the novel method have been revealed.
---
paper_title: Ensembles of wrappers for automated feature selection in fish age classification

paper_content:

A novel feature selection method based on an ensemble of wrappers is proposed.It has been applied for automatically select features in fish age classification.The feature subsets are particularly noticeable given current biological findings.The classification results outperform those based on manual feature selection. In feature selection, the most important features must be chosen so as to decrease the number thereof while retaining their discriminatory information. Within this context, a novel feature selection method based on an ensemble of wrappers is proposed and applied for automatically select features in fish age classification. The effectiveness of this procedure using an Atlantic cod database has been tested for different powerful statistical learning classifiers. The subsets based on few features selected, e.g. otolith weight and fish weight, are particularly noticeable given current biological findings and practices in fishery research and the classification results obtained with them outperforms those of previous studies in which a manual feature selection was performed.
---
paper_title: Sparse Bayesian Learning and the Relevance Vector Machine

paper_content:

This paper introduces a general Bayesian framework for obtaining sparse solutions to regression and classification tasks utilising models linear in the parameters. Although this framework is fully general, we illustrate our approach with a particular specialisation that we denote the 'relevance vector machine' (RVM), a model of identical functional form to the popular and state-of-the-art 'support vector machine' (SVM). We demonstrate that by exploiting a probabilistic Bayesian learning framework, we can derive accurate prediction models which typically utilise dramatically fewer basis functions than a comparable SVM while offering a number of additional advantages. These include the benefits of probabilistic predictions, automatic estimation of 'nuisance' parameters, and the facility to utilise arbitrary basis functions (e.g. non-'Mercer' kernels). We detail the Bayesian framework and associated learning algorithm for the RVM, and give some illustrative examples of its application along with some comparative benchmarks. We offer some explanation for the exceptional degree of sparsity obtained, and discuss and demonstrate some of the advantageous features, and potential extensions, of Bayesian relevance learning.
---
paper_title: The Relevance Sample-Feature Machine: A Sparse Bayesian Learning Approach to Joint Feature-Sample Selection

paper_content:

This paper introduces a novel sparse Bayesian machine-learning algorithm for embedded feature selection in classification tasks. Our proposed algorithm, called the relevance sample feature machine (RSFM), is able to simultaneously choose the relevance samples and also the relevance features for regression or classification problems. We propose a separable model in feature and sample domains. Adopting a Bayesian approach and using Gaussian priors, the learned model by RSFM is sparse in both sample and feature domains. The proposed algorithm is an extension of the standard RVM algorithm, which only opts for sparsity in the sample domain. Experimental comparisons on synthetic as well as benchmark data sets show that RSFM is successful in both feature selection (eliminating the irrelevant features) and accurate classification. The main advantages of our proposed algorithm are: less system complexity, better generalization and avoiding overfitting, and less computational cost during the testing stage.
---
paper_title: Laplacian Score for Feature Selection

paper_content:

In supervised learning scenarios, feature selection has been studied widely in the literature. Selecting features in unsupervised learning scenarios is a much harder problem, due to the absence of class labels that would guide the search for relevant information. And, almost all of previous unsupervised feature selection methods are "wrapper" techniques that require a learning algorithm to evaluate the candidate feature subsets. In this paper, we propose a "filter" method for feature selection which is independent of any learning algorithm. Our method can be performed in either supervised or unsupervised fashion. The proposed method is based on the observation that, in many real world classification problems, data from the same class are often close to each other. The importance of a feature is evaluated by its power of locality preserving, or, Laplacian Score. We compare our method with data variance (unsupervised) and Fisher score (supervised) on two data sets. Experimental results demonstrate the effectiveness and efficiency of our algorithm.
---
paper_title: Floating search methods in feature selection

paper_content:

Sequential search methods characterized by a dynamically changing number of features included or eliminated at each step, henceforth "floating" methods, are presented. They are shown to give very good results and to be computationally more effective than the branch and bound method.
---
paper_title: Using mutual information for selecting features in supervised neural net learning

paper_content:

This paper investigates the application of the mutual information criterion to evaluate a set of candidate features and to select an informative subset to be used as input data for a neural network classifier. Because the mutual information measures arbitrary dependencies between random variables, it is suitable for assessing the "information content" of features in complex classification tasks, where methods bases on linear relations (like the correlation) are prone to mistakes. The fact that the mutual information is independent of the coordinates chosen permits a robust estimation. Nonetheless, the use of the mutual information for tasks characterized by high input dimensionality requires suitable approximations because of the prohibitive demands on computation and samples. An algorithm is proposed that is based on a "greedy" selection of the features and that takes both the mutual information with respect to the output class and with respect to the already-selected features into account. Finally the results of a series of experiments are discussed. >
---
paper_title: Unsupervised feature selection using nonnegative spectral analysis

paper_content:

In this paper, a new unsupervised learning algorithm, namely Nonnegative Discriminative Feature Selection (NDFS), is proposed. To exploit the discriminative information in unsupervised scenarios, we perform spectral clustering to learn the cluster labels of the input samples, during which the feature selection is performed simultaneously. The joint learning of the cluster labels and feature selection matrix enables NDFS to select the most discriminative features. To learn more accurate cluster labels, a nonnegative constraint is explicitly imposed to the class indicators. To reduce the redundant or even noisy features, l2,1-norm minimization constraint is added into the objective function, which guarantees the feature selection matrix sparse in rows. Our algorithm exploits the discriminative information and feature correlation simultaneously to select a better feature subset. A simple yet efficient iterative algorithm is designed to optimize the proposed objective function. Experimental results on different real world datasets demonstrate the encouraging performance of our algorithm over the state-of-the-arts.
---
paper_title: Measuring statistical dependence with Hilbert-Schmidt norms

paper_content:

We propose an independence criterion based on the eigenspectrum of covariance operators in reproducing kernel Hilbert spaces (RKHSs), consisting of an empirical estimate of the Hilbert-Schmidt norm of the cross-covariance operator (we term this a Hilbert-Schmidt Independence Criterion, or HSIC). This approach has several advantages, compared with previous kernel-based independence criteria. First, the empirical estimate is simpler than any other kernel dependence test, and requires no user-defined regularisation. Second, there is a clearly defined population quantity which the empirical estimate approaches in the large sample limit, with exponential convergence guaranteed between the two: this ensures that independence tests based on HSIC do not suffer from slow learning rates. Finally, we show in the context of independent component analysis (ICA) that the performance of HSIC is competitive with that of previously published kernel-based criteria, and of other recently published ICA methods.
---
paper_title: Helmholtz principle based supervised and unsupervised feature selection methods for text mining

paper_content:

We propose new supervised and unsupervised feature selection methods, called Meaning Based Feature Selection (MBFS), for feature selection in text classification.We adapt and use the meaning measure as a new method for feature selection.Meaning measure is based on the Helmholtz principle from the Gestalt theory of human perception.MBFS methods are compared with nine different and well-known feature selection methods on six different datasets.Experimental results show that MBFS methods are effective feature selection methods and have higher speed than several widely used feature selection methods. One of the important problems in text classification is the high dimensionality of the feature space. Feature selection methods are used to reduce the dimensionality of the feature space by selecting the most valuable features for classification. Apart from reducing the dimensionality, feature selection methods have potential to improve text classifiers' performance both in terms of accuracy and time. Furthermore, it helps to build simpler and as a result more comprehensible models. In this study we propose new methods for feature selection from textual data, called Meaning Based Feature Selection (MBFS) which is based on the Helmholtz principle from the Gestalt theory of human perception which is used in image processing. The proposed approaches are extensively evaluated by their effect on the classification performance of two well-known classifiers on several datasets and compared with several feature selection algorithms commonly used in text mining. Our results demonstrate the value of the MBFS methods in terms of classification accuracy and execution time.
---
paper_title: Supervised feature selection via dependence estimation

paper_content:

We introduce a framework for filtering features that employs the Hilbert-Schmidt Independence Criterion (HSIC) as a measure of dependence between the features and the labels. The key idea is that good features should maximise such dependence. Feature selection for various supervised learning problems (including classification and regression) is unified under this framework, and the solutions can be approximated using a backward-elimination algorithm. We demonstrate the usefulness of our method on both artificial and real world datasets.
---
paper_title: Thirteen Ways to Look at the Correlation Coefficient

paper_content:

Abstract In 1885, Sir Francis Galton first defined the term “regression” and completed the theory of bivariate correlation. A decade later, Karl Pearson developed the index that we still use to measure correlation, Pearson's r. Our article is written in recognition of the 100th anniversary of Galton's first discussion of regression and correlation. We begin with a brief history. Then we present 13 different formulas, each of which represents a different computational and conceptual definition of r. Each formula suggests a different way of thinking about this index, from algebraic, geometric, and trigonometric settings. We show that Pearson's r (or simple functions of r) may variously be thought of as a special type of mean, a special type of variance, the ratio of two means, the ratio of two variances, the slope of a line, the cosine of an angle, and the tangent to an ellipse, and may be looked at from several other interesting perspectives.
---
paper_title: Multiclass Linear Dimension Reduction by Weighted Pairwise Fisher Criteria

paper_content:

We derive a class of computationally inexpensive linear dimension reduction criteria by introducing a weighted variant of the well-known K-class Fisher criterion associated with linear discriminant analysis (LDA). It can be seen that LDA weights contributions of individual class pairs according to the Euclidean distance of the respective class means. We generalize upon LDA by introducing a different weighting function.
---
paper_title: Generalized Fisher Score for Feature Selection

paper_content:

Fisher score is one of the most widely used supervised feature selection methods. However, it selects each feature independently according to their scores under the Fisher criterion, which leads to a suboptimal subset of features. In this paper, we present a generalized Fisher score to jointly select features. It aims at finding an subset of features, which maximize the lower bound of traditional Fisher score. The resulting feature selection problem is a mixed integer programming, which can be reformulated as a quadratically constrained linear programming (QCLP). It is solved by cutting plane algorithm, in each iteration of which a multiple kernel learning problem is solved alternatively by multivariate ridge regression and projected gradient descent. Experiments on benchmark data sets indicate that the proposed method outperforms Fisher score as well as many other state-of-the-art feature selection methods.
---
paper_title: Trace ratio criterion for feature selection

paper_content:

Fisher score and Laplacian score are two popular feature selection algorithms, both of which belong to the general graph-based feature selection framework. In this framework, a feature subset is selected based on the corresponding score (subset-level score), which is calculated in a trace ratio form. Since the number of all possible feature subsets is very huge, it is often prohibitively expensive in computational cost to search in a brute force manner for the feature subset with the maximum subset-level score. Instead of calculating the scores of all the feature subsets, traditional methods calculate the score for each feature, and then select the leading features based on the rank of these feature-level scores. However, selecting the feature subset based on the feature-level score cannot guarantee the optimum of the subset-level score. In this paper, we directly optimize the subset-level score, and propose a novel algorithm to efficiently find the global optimal feature subset such that the subset-level score is maximized. Extensive experiments demonstrate the effectiveness of our proposed algorithm in comparison with the traditional methods for feature selection.
---
paper_title: Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy

paper_content:

Feature selection is an important problem for pattern classification systems. We study how to select good features according to the maximal statistical dependency criterion based on mutual information. Because of the difficulty in directly implementing the maximal dependency condition, we first derive an equivalent form, called minimal-redundancy-maximal-relevance criterion (mRMR), for first-order incremental feature selection. Then, we present a two-stage feature selection algorithm by combining mRMR and other more sophisticated feature selectors (e.g., wrappers). This allows us to select a compact set of superior features at very low cost. We perform extensive experimental comparison of our algorithm and other methods using three different classifiers (naive Bayes, support vector machine, and linear discriminate analysis) and four different data sets (handwritten digits, arrhythmia, NCI cancer cell lines, and lymphoma tissues). The results confirm that mRMR leads to promising improvement on feature selection and classification accuracy.
---
paper_title: Adaptive structure learning for low-rank supervised feature selection

paper_content:

Abstract Previous Spectral Feature Selection (SFS) methods output promising feature selection results in many real-world applications, which deeply depend on the preservation of the local or global structures of the data via learning a graph matrix. However, current SFS methods 1) learn the graph matrix in the original data which may contain a number of noise to affect the results of feature selection, 2) conduct the learning of a low-dimensional feature space and the graph matrix individually, thus hard achieve the optimal results of feature selection even though both of these two steps achieve their individual optimization, and 3) consider either the local or global structure of data to difficult provide complementary information for feature selection. To address the above issues, this paper proposes a novel supervised feature selection algorithm to simultaneously preserve the local structure (via adaptive structure learning in a low-dimensional feature space of the original data) and the global structure (via a low-rank constraint) of the data. Moreover, we also propose a new optimization method to fast optimize the resulting objective function. We finally verify the proposed method on eight real-word and benchmark datasets, by comparing with the state-of-the-art feature selection methods, and experimental results show that our proposed method achieves competitive results in term of classification performance.
---
paper_title: The L2,1-norm-based unsupervised optimal feature selection with applications to action recognition

paper_content:

This paper proposes a matrix-based feature selection and classification method that takes the advantage of L2,1-norm regularization. Current studies show that feature extraction and selection have been important steps in classification. However, the existing methods consider feature extraction and selection to be separated phases, which generates suboptimal features for the recognition task. Aiming at making up for this deficiency, we designed a novel classification framework that performs unsupervised optimal feature selection (UOFS) to simultaneously integrate dimensionality reduction, sparse representation, jointly sparse feature extraction and feature selection as well as classification into a unified optimization objective. Specifically, an L2,1-norm-based sparse representation model is constructed as an initial prototype of the proposed method. Then a projection matrix with L2,1-norm regularization is introduced into the model for subspace learning and jointly sparse feature extraction and selection. Finally, we impose a scatter matrix-like constraint on the proposed model in pursuit of the features with less redundancy for recognition. We also provide an alternative iteration optimization with convergence analysis for solving UOFS. Experiments on public gesture and human action datasets validate the superiority of UOFS over other state-of-the-art methods. A feature selection and classification model is proposed.Adjacency graph and projection matrix are simultaneously optimized in our model.The model can conduct jointly sparse feature extraction and feature selection.The high correlation lying in the video frames is reduced in our model.An efficiency algorithm with convergence proof is provided to solve the model.
---
paper_title: Integration of dense subgraph finding with feature clustering for unsupervised feature selection

paper_content:

In this article a dense subgraph finding approach is adopted for the unsupervised feature selection problem. The feature set of a data is mapped to a graph representation with individual features constituting the vertex set and inter-feature mutual information denoting the edge weights. Feature selection is performed in a two-phase approach where the densest subgraph is first obtained so that the features are maximally non-redundant among each other. Finally, in the second stage, feature clustering around the non-redundant features is performed to produce the reduced feature set. An approximation algorithm is used for the densest subgraph finding. Empirically, the proposed approach is found to be competitive with several state of art unsupervised feature selection algorithms.
---
paper_title: Laplacian Score for Feature Selection

paper_content:

In supervised learning scenarios, feature selection has been studied widely in the literature. Selecting features in unsupervised learning scenarios is a much harder problem, due to the absence of class labels that would guide the search for relevant information. And, almost all of previous unsupervised feature selection methods are "wrapper" techniques that require a learning algorithm to evaluate the candidate feature subsets. In this paper, we propose a "filter" method for feature selection which is independent of any learning algorithm. Our method can be performed in either supervised or unsupervised fashion. The proposed method is based on the observation that, in many real world classification problems, data from the same class are often close to each other. The importance of a feature is evaluated by its power of locality preserving, or, Laplacian Score. We compare our method with data variance (unsupervised) and Fisher score (supervised) on two data sets. Experimental results demonstrate the effectiveness and efficiency of our algorithm.
---
paper_title: Unsupervised spectral feature selection with l1-norm graph

paper_content:

Feature selection, which aims to reduce redundancy or noise in the original feature sets, plays an important role in many applications, such as machine learning, multimedia analysis and data mining. Spectral feature selection, a recently proposed method, makes use of spectral clustering to capture underlying manifold structure and achieves excellent performance. However, existing Spectral feature selections suffer from imposing kinds of constraints and lack of clear manifold structure. To address this problem, we propose a new Unsupervised Spectral Feature Selection with l1-Norm Graph, namely USFS. Different from most state-of-art algorithms, the proposed algorithm performs the spectral clustering and l1-Norm Graph jointly to select discriminative features. The manifold structure of original datasets is first learned by the spectral clustering from unlabeled samples, and then it is used to guide the feature selection procedure. Moreover, l1-Norm Graph is imposed to capture clear manifold structure. We also present an efficient iterative optimize method and theoretical convergence analysis of the proposed algorithm. Extensive experimental results on real-world datasets demonstrate the performance of the proposed algorithm.
---
paper_title: Unsupervised feature selection via low-rank approximation and structure learning

paper_content:

Feature selection is an important research topic in machine learning and computer vision in that it can reduce the dimensionality of input data and improve the performance of learning algorithms. Low-rank approximation techniques can well exploit the low-rank property of input data, which coincides with the internal consistency of dimensionality reduction. In this paper, we propose an efficient unsupervised feature selection algorithm, which incorporates low-rank approximation as well as structure learning. First, using the self-representation of data matrix, we formalize the feature selection problem as a matrix factorization with low-rank constraints. This matrix factorization formulation also embeds structure learning regularization as well as a sparse regularized term. Second, we present an effective technique to approximate low-rank constraints and propose a convergent algorithm in a batch mode. This technique can serve as an algorithmic framework for general low-rank recovery problems as well. Finally, the proposed algorithm is validated in twelve publicly available datasets from machine learning repository. Extensive experimental results demonstrate that the proposed method is capable to achieve competitive performance compared to existing state-of-the-art feature selection methods in terms of clustering performance.
---
paper_title: Unsupervised feature selection for multi-cluster data

paper_content:

In many data analysis tasks, one is often confronted with very high dimensional data. Feature selection techniques are designed to find the relevant feature subset of the original features which can facilitate clustering, classification and retrieval. In this paper, we consider the feature selection problem in unsupervised learning scenario, which is particularly difficult due to the absence of class labels that would guide the search for relevant information. The feature selection problem is essentially a combinatorial optimization problem which is computationally expensive. Traditional unsupervised feature selection methods address this issue by selecting the top ranked features based on certain scores computed independently for each feature. These approaches neglect the possible correlation between different features and thus can not produce an optimal feature subset. Inspired from the recent developments on manifold learning and L1-regularized models for subset selection, we propose in this paper a new approach, called Multi-Cluster Feature Selection (MCFS), for unsupervised feature selection. Specifically, we select those features such that the multi-cluster structure of the data can be best preserved. The corresponding optimization problem can be efficiently solved since it only involves a sparse eigen-problem and a L1-regularized least squares problem. Extensive experimental results over various real-life data sets have demonstrated the superiority of the proposed algorithm.
---
paper_title: Robust unsupervised feature selection via matrix factorization

paper_content:

We proposed a robust unsupervised method to remove redundant and irrelevant features.Both the cluster centers and the sparse representation are predicted.The feature selection and clustering are performed simultaneously.An efficient iterative update algorithm based on ADMM is proposed.Superiority over seven existing methods is established for five data sets. Dimensionality reduction is a challenging task for high-dimensional data processing in machine learning and data mining. It can help to reduce computation time, save storage space and improve the performance of learning algorithms. As an effective dimension reduction technique, unsupervised feature selection aims at finding a subset of features to retain the most relevant information. In this paper, we propose a novel unsupervised feature selection method, called Robust Unsupervised Feature Selection via Matrix Factorization (RUFSM), in which robust discriminative feature selection and robust clustering are performed simultaneously under l2, 1-norm while the local manifold structures of data are preserved. The advantages of this work are three-fold. Firstly, both the latent orthogonal cluster centers and the sparse representation of the projected data points based on matrix factorization are predicted for selecting robust discriminative features. Secondly, the feature selection and the clustering are performed simultaneously to guarantee an overall optimum. Thirdly, an efficient iterative update algorithm, which is based on Alternating Direction Method of Multipliers (ADMM), is used for RUFSM optimization. Compared with several state-of-the-art unsupervised feature selection methods, the proposed algorithm comes with better clustering performance for almost all datasets we have experimented with here.
---
paper_title: Unsupervised feature selection by regularized matrix factorization

paper_content:

Abstract Feature selection is an interesting and challenging task in data analysis process. In this paper, a novel algorithm named Regularized Matrix Factorization Feature Selection (RMFFS) is proposed for unsupervised feature selection. Compared with other matrix factorization based feature selection methods, a main advantage of our algorithm is that it takes the correlation among features into consideration. Through introducing an inner product regularization into our algorithm, the features selected by RMFFS would not only well represent the original high-dimensional data, but also contain low redundancy. Moreover, a simple yet efficient iteratively updating algorithm is also developed to solve the proposed RMFFS. Extensive experimental results on nine real world databases demonstrate that our proposed method can achieve better performance than some state-of-the-art unsupervised feature selection methods.
---
paper_title: Unsupervised feature selection by regularized self-representation

paper_content:

By removing the irrelevant and redundant features, feature selection aims to find a compact representation of the original feature with good generalization ability. With the prevalence of unlabeled data, unsupervised feature selection has shown to be effective in alleviating the curse of dimensionality, and is essential for comprehensive analysis and understanding of myriads of unlabeled high dimensional data. Motivated by the success of low-rank representation in subspace clustering, we propose a regularized self-representation (RSR) model for unsupervised feature selection, where each feature can be represented as the linear combination of its relevant features. By using L 2 , 1 -norm to characterize the representation coefficient matrix and the representation residual matrix, RSR is effective to select representative features and ensure the robustness to outliers. If a feature is important, then it will participate in the representation of most of other features, leading to a significant row of representation coefficients, and vice versa. Experimental analysis on synthetic and real-world data demonstrates that the proposed method can effectively identify the representative features, outperforming many state-of-the-art unsupervised feature selection methods in terms of clustering accuracy, redundancy reduction and classification accuracy. HighlightsA regularized self-representation (RSR) model is proposed for unsupervised feature selection.An iterative reweighted least-squares algorithm is proposed to solve the RSR model.The proposed method shows superior performance to state-of-the-art.
---
paper_title: Unsupervised Feature Selection Using Feature Similarity

paper_content:

In this article, we describe an unsupervised feature selection algorithm suitable for data sets, large in both dimension and size. The method is based on measuring similarity between features whereby redundancy therein is removed. This does not need any search and, therefore, is fast. A new feature similarity measure, called maximum information compression index, is introduced. The algorithm is generic in nature and has the capability of multiscale representation of data sets. The superiority of the algorithm, in terms of speed and performance, is established extensively over various real-life data sets of different sizes and dimensions. It is also demonstrated how redundancy and information loss in feature selection can be quantified with an entropy measure.
---
paper_title: Spectral feature selection for supervised and unsupervised learning

paper_content:

Feature selection aims to reduce dimensionality for building comprehensible learning models with good generalization performance. Feature selection algorithms are largely studied separately according to the type of learning: supervised or unsupervised. This work exploits intrinsic properties underlying supervised and unsupervised feature selection algorithms, and proposes a unified framework for feature selection based on spectral graph theory. The proposed framework is able to generate families of algorithms for both supervised and unsupervised feature selection. And we show that existing powerful algorithms such as ReliefF (supervised) and Laplacian Score (unsupervised) are special cases of the proposed framework. To the best of our knowledge, this work is the first attempt to unify supervised and unsupervised feature selection, and enable their joint study under a general framework. Experiments demonstrated the efficacy of the novel algorithms derived from the framework.
---
paper_title: Unsupervised feature selection via Diversity-induced Self-representation

paper_content:

Feature selection is to select a subset of relevant features from the original feature set. In practical applications, regarding the unavailability of an amount of the labels is still a challenging problem. To overcome this problem, unsupervised feature selection algorithms have been developed and achieve promising performance. However, most existing approaches consider only the representativeness of features, but the diversity of features which may lead to the high redundancy and the losses of valuable features are ignored. In this paper, we propose a Diversity-induced Self-representation (DISR) based unsupervised feature selection method to effectively select the features with both representativeness and diversity. Specifically, based on the inherent self-representation property of features, the most representative features can be selected. Meanwhile, to preserve the diversity of selected features and reduce the redundancy of the original features as soon as possible, we introduce a novel diversity term, which adjusts the weights of selected features by incorporating the similarities between features. We then present an efficient algorithm to solve the optimization problem by using the inexact Augmented Lagrange Method (ALM). Finally, both clustering and classification tasks are used to evaluate the proposed method. Empirical results on the synthetic dataset and nine real-world datasets demonstrate the superiority of our method compared with state-of-the-art algorithms.
---
paper_title: Graph self-representation method for unsupervised feature selection

paper_content:

Abstract Both subspace learning methods and feature selection methods are often used for removing irrelative features from high-dimensional data. Studies have shown that feature selection methods have interpretation ability and subspace learning methods output stable performance. This paper proposes a new unsupervised feature selection by integrating a subspace learning method (i.e., Locality Preserving Projection (LPP)) into a new feature selection method (i.e., a sparse feature-level self-representation method), aim at simultaneously receiving stable performance and interpretation ability. Different from traditional sample-level self-representation where each sample is represented by all samples and has been popularly used in machine learning and computer vision. In this paper, we propose to represent each feature by its relevant features to conduct feature selection via devising a feature-level self-representation loss function plus an l 2 , 1 -norm regularization term. Then we add a graph regularization term (i.e., LPP) into the resulting feature selection model to simultaneously conduct feature selection and subspace learning. The rationale of the LPP regularization term is that LPP preserves the original distribution of data after removing irrelative features. Finally, we conducted experiments on UCI data sets and other real data sets and the experimental results showed that the proposed approach outperformed all comparison algorithms.
---
paper_title: Spectral Embedded Clustering: A Framework for In-Sample and Out-of-Sample Spectral Clustering

paper_content:

Spectral clustering (SC) methods have been successfully applied to many real-world applications. The success of these SC methods is largely based on the manifold assumption, namely, that two nearby data points in the high-density region of a low-dimensional data manifold have the same cluster label. However, such an assumption might not always hold on high-dimensional data. When the data do not exhibit a clear low-dimensional manifold structure (e.g., high-dimensional and sparse data), the clustering performance of SC will be degraded and become even worse than K -means clustering. In this paper, motivated by the observation that the true cluster assignment matrix for high-dimensional data can be always embedded in a linear space spanned by the data, we propose the spectral embedded clustering (SEC) framework, in which a linearity regularization is explicitly added into the objective function of SC methods. More importantly, the proposed SEC framework can naturally deal with out-of-sample data. We also present a new Laplacian matrix constructed from a local regression of each pattern and incorporate it into our SEC framework to capture both local and global discriminative information for clustering. Comprehensive experiments on eight real-world high-dimensional datasets demonstrate the effectiveness and advantages of our SEC framework over existing SC methods and K-means-based clustering methods. Our SEC framework significantly outperforms SC using the Nystrom algorithm on unseen data.
---
paper_title: Unsupervised feature selection using nonnegative spectral analysis

paper_content:

In this paper, a new unsupervised learning algorithm, namely Nonnegative Discriminative Feature Selection (NDFS), is proposed. To exploit the discriminative information in unsupervised scenarios, we perform spectral clustering to learn the cluster labels of the input samples, during which the feature selection is performed simultaneously. The joint learning of the cluster labels and feature selection matrix enables NDFS to select the most discriminative features. To learn more accurate cluster labels, a nonnegative constraint is explicitly imposed to the class indicators. To reduce the redundant or even noisy features, l2,1-norm minimization constraint is added into the objective function, which guarantees the feature selection matrix sparse in rows. Our algorithm exploits the discriminative information and feature correlation simultaneously to select a better feature subset. A simple yet efficient iterative algorithm is designed to optimize the proposed objective function. Experimental results on different real world datasets demonstrate the encouraging performance of our algorithm over the state-of-the-arts.
---
paper_title: Semi-supervised Feature Selection via Spectral Analysis

paper_content:

Feature selection is an important task in effective data mining. A new challenge to feature selection is the socalled “small labeled-sample problem” in which labeled data is small and unlabeled data is large. The paucity of labeled instances provides insufficient information about the structure of the target concept, and can cause supervised feature selection algorithms to fail. Unsupervised feature selection algorithms can work without labeled data. However, these algorithms ignore label information, which may lead to performance deterioration. In this work, we propose to use both (small) labeled and (large) unlabeled data in feature selection, which is a topic has not yet been addressed in feature selection research. We present a semi-supervised feature selection algorithm based on spectral analysis. The algorithm exploits both labeled and unlabeled data through a regularization framework, which provides an effective way to address the “small labeled-sample” problem. Experimental results demonstrated the efficacy of our approach and confirmed that small labeled samples can help feature selection with unlabeled data. Keyword: Feature Selection, Semi-supervised Learning, Machine Learning, Spectral Analysis
---
paper_title: Image annotation by kNN-sparse graph-based label propagation over noisily tagged web images

paper_content:

In this article, we exploit the problem of annotating a large-scale image corpus by label propagation over noisily tagged web images. To annotate the images more accurately, we propose a novel kNN-sparse graph-based semi-supervised learning approach for harnessing the labeled and unlabeled data simultaneously. The sparse graph constructed by datum-wise one-vs-kNN sparse reconstructions of all samples can remove most of the semantically unrelated links among the data, and thus it is more robust and discriminative than the conventional graphs. Meanwhile, we apply the approximate k nearest neighbors to accelerate the sparse graph construction without loosing its effectiveness. More importantly, we propose an effective training label refinement strategy within this graph-based learning framework to handle the noise in the training labels, by bringing in a dual regularization for both the quantity and sparsity of the noise. We conduct extensive experiments on a real-world image database consisting of 55,615 Flickr images and noisily tagged training labels. The results demonstrate both the effectiveness and efficiency of the proposed approach and its capability to deal with the noise in the training labels.
---
paper_title: Web and Personal Image Annotation by Mining Label Correlation With Relaxed Visual Graph Embedding

paper_content:

The number of digital images rapidly increases, and it becomes an important challenge to organize these resources effectively. As a way to facilitate image categorization and retrieval, automatic image annotation has received much research attention. Considering that there are a great number of unlabeled images available, it is beneficial to develop an effective mechanism to leverage unlabeled images for large-scale image annotation. Meanwhile, a single image is usually associated with multiple labels, which are inherently correlated to each other. A straightforward method of image annotation is to decompose the problem into multiple independent single-label problems, but this ignores the underlying correlations among different labels. In this paper, we propose a new inductive algorithm for image annotation by integrating label correlation mining and visual similarity mining into a joint framework. We first construct a graph model according to image visual features. A multilabel classifier is then trained by simultaneously uncovering the shared structure common to different labels and the visual graph embedded label prediction matrix for image annotation. We show that the globally optimal solution of the proposed framework can be obtained by performing generalized eigen-decomposition. We apply the proposed framework to both web image annotation and personal album labeling using the NUS-WIDE, MSRA MM 2.0, and Kodak image data sets, and the AUC evaluation metric. Extensive experiments on large-scale image databases collected from the web and personal album show that the proposed algorithm is capable of utilizing both labeled and unlabeled data for image annotation and outperforms other algorithms.
---
paper_title: Discriminating Joint Feature Analysis for Multimedia Data Understanding

paper_content:

In this paper, we propose a novel semi-supervised feature analyzing framework for multimedia data understanding and apply it to three different applications: image annotation, video concept detection and 3-D motion data analysis. Our method is built upon two advancements of the state of the art: (1) l2, 1-norm regularized feature selection which can jointly select the most relevant features from all the data points. This feature selection approach was shown to be robust and efficient in literature as it considers the correlation between different features jointly when conducting feature selection; (2) manifold learning which analyzes the feature space by exploiting both labeled and unlabeled data. It is a widely used technique to extend many algorithms to semi-supervised scenarios for its capability of leveraging the manifold structure of multimedia data. The proposed method is able to learn a classifier for different applications by selecting the discriminating features closely related to the semantic concepts. The objective function of our method is non-smooth and difficult to solve, so we design an efficient iterative algorithm with fast convergence, thus making it applicable to practical applications. Extensive experiments on image annotation, video concept detection and 3-D motion data analysis are performed on different real-world data sets to demonstrate the effectiveness of our algorithm.
---
paper_title: Hyperspectral image classification using Fisher criterion-based Gabor cube selection and multi-task joint sparse representation

paper_content:

Recently, Gabor wavelet transformation has been introduced for feature extraction of hyperspectral imagery. Due to the discriminative power of obtained Gabor features, high classification performance has been achieved. However, thousands of Gabor features cause too much burden for onboard computation, limiting the efficiency of the method. In fact, not all features have a positive effect on classification. In this paper, we have proposed a Gabor cube selection-based Multi-task Joint Sparse Representation framework, abbreviated as MT-SG, for hyperspectral imagery classification. Firstly, based on the Fisher discrimination criterion, the most representative Gabor cubes for each class have been picked out. Next, under multi-task joint sparse representation framework, a coefficient vector can be obtained for each test sample with the selected cube features, which can be applied for the following residual-based classification. Experimental results on real hyperspectral data have demonstrated the feasibility and efficiency of the proposed method.
---
paper_title: Survey and taxonomy of feature selection algorithms in intrusion detection system

paper_content:

The Intrusion detection system deals with huge amount of data which contains irrelevant and redundant features causing slow training and testing process, higher resource consumption as well as poor detection rate. Feature selection, therefore, is an important issue in intrusion detection. In this paper we introduce concepts and algorithms of feature selection, survey existing feature selection algorithms in intrusion detection systems, group and compare different algorithms in three broad categories: filter, wrapper, and hybrid. We conclude the survey by identifying trends and challenges of feature selection research and development in intrusion detection system.
---
paper_title: Mutual information-based feature selection for intrusion detection systems

paper_content:

As the network-based technologies become omnipresent, threat detection and prevention for these systems become increasingly important. One of the effective ways to achieve higher security is to use intrusion detection systems, which are software tools used to detect abnormal activities in the computer or network. One technical challenge in intrusion detection systems is the curse of high dimensionality. To overcome this problem, we propose a feature selection phase, which can be generally implemented in any intrusion detection system. In this work, we propose two feature selection algorithms and study the performance of using these algorithms compared to a mutual information-based feature selection method. These feature selection algorithms require the use of a feature goodness measure. We investigate using both a linear and a non-linear measure-linear correlation coefficient and mutual information, for the feature selection. Further, we introduce an intrusion detection system that uses an improved machine learning based method, Least Squares Support Vector Machine. Experiments on KDD Cup 99 data set address that our proposed mutual information-based feature selection method results in detecting intrusions with higher accuracy, especially for remote to login (R2L) and user to remote (U2R) attacks.
---
paper_title: Tabu Search and Binary Particle Swarm Optimization for Feature Selection Using Microarray Data

paper_content:

Gene expression profiles have great potential as a medical diagnosis tool because they represent the state of a cell at the molecular level. In the classification of cancer type research, available training datasets generally have a fairly small sample size compared to the number of genes involved. This fact poses an unprecedented challenge to some classification methodologies due to training data limitations. Therefore, a good selection method for genes relevant for sample classification is needed to improve the predictive accuracy, and to avoid incomprehensibility due to the large number of genes investigated. In this article, we propose to combine tabu search (TS) and binary particle swarm optimization (BPSO) for feature selection. BPSO acts as a local optimizer each time the TS has been run for a single generation. The K-nearest neighbor method with leave-one-out cross-validation and support vector machine with one-versus-rest serve as evaluators of the TS and BPSO. The proposed method is applied and compared to the 11 classification problems taken from the literature. Experimental results show that our method simplifies features effectively and either obtains higher classification accuracy or uses fewer features compared to other feature selection methods.
---
paper_title: An Improved Minimum Redundancy Maximum Relevance Approach for Feature Selection in Gene Expression Data

paper_content:

Abstract In this article, an improved feature selection technique has been proposed. Mutual Information is taken as the basic criterion to find the feature relevance and redundancy. The mutual information between a feature and class labels defines the relevance of that feature. Again, the mutual information among different features defines the correlation i.e., the redundancy among those features. Now our objective is to find such a feature set for which the mutual information among the features and the class labels are maximized and the mutual information among the features are minimized. Therefore, the goal of the proposed method is to find the most relevant and least redundant feature set. The number of output features is provided by the user. First the most relevant feature is added to the empty final feature set. Then in each iteration a non-dominated feature set with respect to relevance and redundancy is generated and from this set of features, the most relevant and non-redundant feature is included in the final feature set. Thereafter, in an incremental way a feature is added in every iteration and this step is repeated while the size of the final feature set is equal to the user given number of features. The features contained by the final feature set have maximum relevance and least correlation. The proposed method is applied on microarray gene expression data to find the most relevant and non-redundant genes and the performance of the proposed method is compared with that of the popular mRMR (MIQ) and mRMR (MID) schemes on several real-life data sets.
---
paper_title: Detection of subjects and brain regions related to Alzheimer's disease using 3D MRI scans based on eigenbrain and machine learning

paper_content:

(Purpose) Early diagnosis or detection of Alzheimer’s disease (AD) from the normal elder control (NC) is very important. However, the computer-aided diagnosis (CAD) was not widely used, and the classification performance did not reach the standard of practical use. We proposed a novel CAD system for MR brain images based on eigenbrains and machine learning with two goals: accurate detection of both AD subjects and AD-related brain regions. (Method) First, we used maximum inter-class variance to select key slices from 3D volumetric data. Second, we generated an eigenbrain set for each subject. Third, the most important eigenbrain (MIE) was obtained by Welch’s t-test. Finally, kernel support-vector-machines with different kernels that were trained by particle swarm optimization, were used to make an accurate prediction of AD subjects. Coefficients of MIE with values higher than 0.98 quantile were highlighted to obtain the discriminant regions that distinguish AD from NC. (Results) The experiments showed that the proposed method can predict AD subjects with a competitive performance with existing methods, especially the accuracy of the polynomial kernel (92.36±0.94) was better than the linear kernel of 91.47±1.02 and the radial basis function (RBF) kernel of 86.71±1.93. The proposed eigenbrain-based CAD system detected 30 AD-related brain regions. The results were coherent with existing literatures. (Conclusion) The eigenbrain method was effective in AD subject prediction and discriminant brain-region detection in MRI scanning.
---
paper_title: Detection of Life-Threatening Arrhythmias Using Feature Selection and Support Vector Machines

paper_content:

Early detection of ventricular fibrillation (VF) and rapid ventricular tachycardia (VT) is crucial for the success of the defibrillation therapy. A wide variety of detection algorithms have been proposed based on temporal, spectral, or complexity parameters extracted from the ECG. However, these algorithms are mostly constructed by considering each parameter individually. In this study, we present a novel life-threatening arrhythmias detection algorithm that combines a number of previously proposed ECG parameters by using support vector machines classifiers. A total of 13 parameters were computed accounting for temporal (morphological), spectral, and complexity features of the ECG signal. A filter-type feature selection (FS) procedure was proposed to analyze the relevance of the computed parameters and how they affect the detection performance. The proposed methodology was evaluated in two different binary detection scenarios: shockable (FV plus VT) versus nonshockable arrhythmias, and VF versus nonVF rhythms, using the information contained in the medical imaging technology database, the Creighton University ventricular tachycardia database, and the ventricular arrhythmia database. sensitivity (SE) and specificity (SP) analysis on the out of sample test data showed values of SE=95%, SP=99%, and SE=92% , SP=97% in the case of shockable and VF scenarios, respectively. Our algorithm was benchmarked against individual detection schemes, significantly improving their performance. Our results demonstrate that the combination of ECG parameters using statistical learning algorithms improves the efficiency for the detection of life-threatening arrhythmias.
---

Title: A Review of Feature Selection and Its Methods

Section 1: Introduction
Description 1: This section introduces the background and significance of feature selection in handling large datasets with noisy, redundant, and irrelevant data, and discusses key concepts and terminologies.

Section 2: FS procedure
Description 2: This section outlines the procedural steps involved in the feature selection process, including search direction, search strategies, and evaluation criteria.

Section 3: Feature selection based on Evaluation criteria
Description 3: This section discusses feature selection algorithms categorized based on their evaluation criteria, including filter, wrapper, and embedded methods.

Section 4: Filter method
Description 4: This section provides an in-depth explanation of the filter method, including statistical measures used to select the best subset of features.

Section 5: Wrapper method
Description 5: This section details the wrapper method, discussing its process and various algorithms used to select optimal feature subsets.

Section 6: Embedded method
Description 6: This section covers the embedded method, highlighting how feature selection is performed during the model learning process to improve computational cost and classification accuracy.

Section 7: Findings
Description 7: This section summarizes the findings from the literature review, addressing the strengths and weaknesses of different feature selection methods.

Section 8: FS methods based on Learning method
Description 8: This section categorizes feature selection methods based on supervised, unsupervised, and semi-supervised learning models.

Section 9: Applications
Description 9: This section examines the applications of feature selection in various domains such as hyperspectral images, intrusion detection, and microarray data.

Section 10: Conclusion and future scope
Description 10: This section provides a conclusion of the survey and discusses potential future research directions in feature selection, focusing on hybrid methods, genetic algorithm-based searching methods, and dynamic datasets.