Causal Feature Selection via Transfer Entropy
Paolo Bonetti Alberto Maria Metelli Marcello Restelli
Machine learning algorithms are designed to capture complex relationships
between features. In this context, the high dimensionality of data often
results in poor model performance, with the risk of overfitting. Feature
selection, the process of selecting a subset of relevant and non-redundant
features, is, therefore, an essential step to mitigate these issues. However,
classical feature selection approaches do not inspect the causal relationship
between selected features and target, which can lead to misleading results in
real-world applications. Causal discovery, instead, aims to identify causal
relationships between features with observational data. In this paper, we
propose a novel methodology at the intersection between feature selection and
causal discovery, focusing on time series. We introduce a new causal feature
selection approach that relies on the forward and backward feature selection
procedures and leverages transfer entropy to estimate the causal flow of
information from the features to the target in time series. Our approach
enables the selection of features not only in terms of mere model performance
but also captures the causal information flow. In this context, we provide
theoretical guarantees on the regression and classification errors for both the
exact and the finite-sample cases. Finally, we present numerical validations on
synthetic and real-world regression problems, showing results competitive
w.r.t. the considered baselines.