The paper, interested in unsupervised feature selection, aims to retain the features best accounting for the local patterns in the data. The proposed approach, called Locally Linear Unsupervised Feature Selection, relies on a dimensionality reduction method to characterize such patterns; each feature is thereafter assessed according to its compliance w.r.t. the local patterns, taking inspiration from Locally Linear Embedding (Roweis and Saul, 2000). The experimental validation of the approach on the scikit-feature benchmark suite demonstrates its effectiveness compared to the state of the art. Machine Learning faces statistical and computational challenges due to the increasing dimension of modern datasets. Dimensionality reduction aims at addressing such challenges through embedding the data in a lower dimensionality space, in an unsupervised BID17 BID28 BID34 BID41 or supervised BID9 BID8 BID25 way.The requirement for understandable Machine Learning BID36 BID5 however makes it desirable to achieve interpretable dimensionality reduction. In order to do so, the simplest way is to select a subset of the initial features, i.e. to achieve feature selection (FS), as opposed to generating compound new features from the initial ones, a.k.a. feature construction. For instance, determining the genes most important w.r.t. a given disease or the underlying generative model of the data can be viewed as the mother goal in bioinformatics BID13 BID23 .In the supervised ML setting, features are assessed and selected based on their relevance to the prediction goal BID11 BID31 BID3 . Unsupervised learning, aimed at making sense of the data, however constitutes a primary and most important task of ML, as emphasized by BID20 , while supervised ML intervenes at a later stage of the data exploitation process.Unsupervised FS approaches BID14 BID47 BID2 BID22 BID48 ) (more in section 2) essentially rely on the assumption that the data samples are structured in clusters, and use the cluster partition in lieu of labels, making it possible to fall down on supervised FS, and select the features most amenable to characterize and separate the clusters. A main limitation of this methodology is that clusters are bound to rely on some metric defined from the initial features (with the notable exception of BID22 ), although this metric can be arbitrarily corrupted based on irrelevant or random features. On the other hand , as far as one considers the unsupervised setting, a feature can hardly be considered irrelevant per se.The main contribution of the paper is to address both limitations: the proposed approach, called Locally Linear Unsupervised Feature Selection (LLUFS) jointly determines patterns in the data, and features relevant to characterize these patterns. LLUFS is a 2-step process (Sec. 3): In a first step, a compressed representation of the data is built using Auto-Encoders BID37 BID7 . In a second step, viewing the initial dataset as a high-dimensional embedding of the compressed dataset, each feature is scored according to its contribution to the reconstruction error of the embedding, taking inspiration from Locally Linear Embedding BID28 BID29 BID40 .After describing the goals of experiments and the experimental setting used to validate the approach, extensively relying on the scikit-feature project BID21 SKf, 2018) (Sec. 4) , the empirical validation is presented and discussed (Sec. 5), establishing the merits and discussing the weaknesses of the approach. The paper concludes with a discussion and some perspectives for further research. One weakness of the method is that the distorsion scores depend on the latent representation produced by the auto-encoder, which might be biased due to the redundancy of the initial features; typically, duplicating an initial feature will entail that the latent representation is more able to express this feature, mechanically reducing its distorsion score. For this reason, a preliminary step is to detect and reduce the redundancy of the initial features.In order to do so, LLUFS i) normalizes the initial features (with zero mean and unit variance); ii) uses Agglomerative Hierarchical feature clustering BID18 BID35 , using a high number of clusters n c (n c = 3 4 D in the experiments); iii) selects one feature per cluster (the nearest one to the cluster mean); iv) apply the auto-encoder on the pruned data.Further work is concerned with taking into account the feature redundancy within the AE loss.A second limitation is due to the sensitivity of the distorsion score to the feature distribution. Typically, while a constant feature carries no information, its distorsion is null. Likewise, the distorsion of discrete features depends on their being balanced. In order to alleviate this issue, the reliability of the distorsion associated to each feature is measured through an empirical p-value BID33 . Given a p-value threshold τ , 1/τ copies of each feature are generated and independently shuffled. The feature distorsion is deemed relevant iff it is lower than the distorsion of all shuffled copies. A novel approach to unsupervised feature selection has been proposed in this paper, with a proof of concept of its empirical merits. The core idea is to find an "oracle" representation of the data, and to consider the actual data as an inflated and corrupted image of the oracle data. The quality of each feature is thereafter assessed depending on how it contributes to the loss of information between the "oracle" and the actual data.A first perspective for further research, taking inspiration from NDFS, is to allow a feature to be partially relevant, e.g. through considering the quantiles of its distorsion. A second perspective is to integrate the feature redundancy in the auto-encoder loss, to decrease the bias in favor of redundant features. The approach will also be extended to supervised feature selection.
Unsupervised feature selection through capturing the local linear structure of the data Proposes locally linear unsupervised feature selection. The paper proposes the LLUFS method for feature selection.