Winning Prize Comes from Losing Tickets: Improve Invariant Learning by
  Exploring Variant Parameters for Out-of-Distribution Generalization
Zhuo Huang Muyang Li Li Shen Jun Yu Chen Gong Bo Han Tongliang Liu
Out-of-Distribution (OOD) Generalization aims to learn robust models that
generalize well to various environments without fitting to
distribution-specific features. Recent studies based on Lottery Ticket
Hypothesis (LTH) address this problem by minimizing the learning target to find
some of the parameters that are critical to the task. However, in OOD problems,
such solutions are suboptimal as the learning task contains severe distribution
noises, which can mislead the optimization process. Therefore, apart from
finding the task-related parameters (i.e., invariant parameters), we propose
Exploring Variant parameters for Invariant Learning (EVIL) which also leverages
the distribution knowledge to find the parameters that are sensitive to
distribution shift (i.e., variant parameters). Once the variant parameters are
left out of invariant learning, a robust subnetwork that is resistant to
distribution shift can be found. Additionally, the parameters that are
relatively stable across distributions can be considered invariant ones to
improve invariant learning. By fully exploring both variant and invariant
parameters, our EVIL can effectively identify a robust subnetwork to improve
OOD generalization. In extensive experiments on integrated testbed: DomainBed,
EVIL can effectively and efficiently enhance many popular methods, such as ERM,
IRM, SAM, etc.