An information theoretic approach to quantify the stability of feature
  selection and ranking algorithms
Alaiz-Rodriguez R. Parnell A. C
Feature selection is a key step when dealing with high dimensional data. In
particular, these techniques simplify the process of knowledge discovery from
the data by selecting the most relevant features out of the noisy, redundant
and irrelevant features. A problem that arises in many of these practical
applications is that the outcome of the feature selection algorithm is not
stable. Thus, small variations in the data may yield very different feature
rankings. Assessing the stability of these methods becomes an important issue
in the previously mentioned situations. We propose an information theoretic
approach based on the Jensen Shannon divergence to quantify this robustness.
Unlike other stability measures, this metric is suitable for different
algorithm outcomes: full ranked lists, feature subsets as well as the lesser
studied partial ranked lists. This generalized metric quantifies the difference
among a whole set of lists with the same size, following a probabilistic
approach and being able to give more importance to the disagreements that
appear at the top of the list. Moreover, it possesses desirable properties
including correction for change, upper lower bounds and conditions for a
deterministic selection. We illustrate the use of this stability metric with
data generated in a fully controlled way and compare it with popular metrics
including the Spearmans rank correlation and the Kunchevas index on feature
ranking and selection outcomes, respectively. Additionally, experimental
validation of the proposed approach is carried out on a real-world problem of
food quality assessment showing its potential to quantify stability from
different perspectives.