Adversarial Learning for Feature Shift Detection and Correction
Miriam Barrabes Daniel Mas Montserrat Margarita Geleta Xavier Giro-i-Nieto Alexander G. Ioannidis
Data shift is a phenomenon present in many real-world applications, and while
there are multiple methods attempting to detect shifts, the task of localizing
and correcting the features originating such shifts has not been studied in
depth. Feature shifts can occur in many datasets, including in multi-sensor
data, where some sensors are malfunctioning, or in tabular and structured data,
including biomedical, financial, and survey data, where faulty standardization
and data processing pipelines can lead to erroneous features. In this work, we
explore using the principles of adversarial learning, where the information
from several discriminators trained to distinguish between two distributions is
used to both detect the corrupted features and fix them in order to remove the
distribution shift between datasets. We show that mainstream supervised
classifiers, such as random forest or gradient boosting trees, combined with
simple iterative heuristics, can localize and correct feature shifts,
outperforming current statistical and neural network-based techniques. The code
is available at https://github.com/AI-sandbox/DataFix.