Connect Later: Improving Fine-tuning for Robustness with Targeted
  Augmentations
Helen Qu Sang Michael Xie
Models trained on a labeled source domain (e.g., labeled images from wildlife
camera traps) often generalize poorly when deployed on an out-of-distribution
(OOD) target domain (e.g., images from new camera trap locations). In the
domain adaptation setting where unlabeled target data is available,
self-supervised pretraining (e.g., masked autoencoding or contrastive learning)
is a promising method to mitigate this performance drop. Pretraining improves
OOD error when the generic data augmentations used (e.g., masking or cropping)
connect the source and target domains, which may be far apart in the input
space. In this paper, we show on real-world tasks that standard fine-tuning
after pretraining does not consistently improve OOD error over simply training
from scratch on labeled source data. To better leverage pretraining for
distribution shifts, we propose Connect Later: after pretraining with generic
augmentations, fine-tune with targeted augmentations designed with knowledge of
the distribution shift. Pretraining learns good representations within the
source and target domains, while targeted augmentations connect the domains
better during fine-tuning. Connect Later improves average OOD error over
standard fine-tuning and supervised learning with targeted augmentations on 4
real-world datasets: Connect Later achieves the state-of-the-art on
astronomical time-series classification (AstroClassification) by 2.5%, wildlife
species identification (iWildCam-WILDS) with ResNet-50 by 0.9%, and tumor
identification (Camelyon17-WILDS) with DenseNet121 by 1.1%; as well as best
performance on a new dataset for astronomical time-series redshift prediction
(Redshifts) by 0.03 RMSE (11% relative). Code and datasets are available at
https://github.com/helenqu/connect-later.