A Novel Information-Theoretic Objective to Disentangle Representations
  for Fair Classification
Pierre Colombo Nathan Noiry Guillaume Staerman Pablo Piantanida
One of the pursued objectives of deep learning is to provide tools that learn
abstract representations of reality from the observation of multiple contextual
situations. More precisely, one wishes to extract disentangled representations
which are (i) low dimensional and (ii) whose components are independent and
correspond to concepts capturing the essence of the objects under consideration
(Locatello et al., 2019b). One step towards this ambitious project consists in
learning disentangled representations with respect to a predefined (sensitive)
attribute, e.g., the gender or age of the writer. Perhaps one of the main
application for such disentangled representations is fair classification.
Existing methods extract the last layer of a neural network trained with a loss
that is composed of a cross-entropy objective and a disentanglement
regularizer. In this work, we adopt an information-theoretic view of this
problem which motivates a novel family of regularizers that minimizes the
mutual information between the latent representation and the sensitive
attribute conditional to the target. The resulting set of losses, called
CLINIC, is parameter free and thus, it is easier and faster to train. CLINIC
losses are studied through extensive numerical experiments by training over 2k
neural networks. We demonstrate that our methods offer a better
disentanglement/accuracy trade-off than previous techniques, and generalize
better than training with cross-entropy loss solely provided that the
disentanglement task is not too constraining.