Environment Diversification with Multi-head Neural Network for Invariant
  Learning
Bo-Wei Huang Keng-Te Liao Chang-Sheng Kao Shou-De Lin
Neural networks are often trained with empirical risk minimization; however,
it has been shown that a shift between training and testing distributions can
cause unpredictable performance degradation. On this issue, a research
direction, invariant learning, has been proposed to extract invariant features
insensitive to the distributional changes. This work proposes EDNIL, an
invariant learning framework containing a multi-head neural network to absorb
data biases. We show that this framework does not require prior knowledge about
environments or strong assumptions about the pre-trained model. We also reveal
that the proposed algorithm has theoretical connections to recent studies
discussing properties of variant and invariant features. Finally, we
demonstrate that models trained with EDNIL are empirically more robust against
distributional shifts.