Nonlinear Feature Aggregation: Two Algorithms driven by Theory
Paolo Bonetti Alberto Maria Metelli Marcello Restelli
Many real-world machine learning applications are characterized by a huge
number of features, leading to computational and memory issues, as well as the
risk of overfitting. Ideally, only relevant and non-redundant features should
be considered to preserve the complete information of the original data and
limit the dimensionality. Dimensionality reduction and feature selection are
common preprocessing techniques addressing the challenge of efficiently dealing
with high-dimensional data. Dimensionality reduction methods control the number
of features in the dataset while preserving its structure and minimizing
information loss. Feature selection aims to identify the most relevant features
for a task, discarding the less informative ones. Previous works have proposed
approaches that aggregate features depending on their correlation without
discarding any of them and preserving their interpretability through
aggregation with the mean. A limitation of methods based on correlation is the
assumption of linearity in the relationship between features and target. In
this paper, we relax such an assumption in two ways. First, we propose a
bias-variance analysis for general models with additive Gaussian noise, leading
to a dimensionality reduction algorithm (NonLinCFA) which aggregates non-linear
transformations of features with a generic aggregation function. Then, we
extend the approach assuming that a generalized linear model regulates the
relationship between features and target. A deviance analysis leads to a second
dimensionality reduction algorithm (GenLinCFA), applicable to a larger class of
regression problems and classification settings. Finally, we test the
algorithms on synthetic and real-world datasets, performing regression and
classification tasks, showing competitive performances.