Benign Shortcut for Debiasing: Fair Visual Recognition via Intervention
  with Shortcut Features
Yi Zhang Jitao Sang Junyang Wang Dongmei Jiang Yaowei Wang
Machine learning models often learn to make predictions that rely on
sensitive social attributes like gender and race, which poses significant
fairness risks, especially in societal applications, such as hiring, banking,
and criminal justice. Existing work tackles this issue by minimizing the
employed information about social attributes in models for debiasing. However,
the high correlation between target task and these social attributes makes
learning on the target task incompatible with debiasing. Given that model bias
arises due to the learning of bias features (\emph{i.e}., gender) that help
target task optimization, we explore the following research question: \emph{Can
we leverage shortcut features to replace the role of bias feature in target
task optimization for debiasing?} To this end, we propose \emph{Shortcut
Debiasing}, to first transfer the target task's learning of bias attributes
from bias features to shortcut features, and then employ causal intervention to
eliminate shortcut features during inference. The key idea of \emph{Shortcut
Debiasing} is to design controllable shortcut features to on one hand replace
bias features in contributing to the target task during the training stage, and
on the other hand be easily removed by intervention during the inference stage.
This guarantees the learning of the target task does not hinder the elimination
of bias features. We apply \emph{Shortcut Debiasing} to several benchmark
datasets, and achieve significant improvements over the state-of-the-art
debiasing methods in both accuracy and fairness.