Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift
Benjamin Eyre Elliot Creager David Madras Vardan Papyan Richard Zemel
Designing deep neural network classifiers that perform robustly on
distributions differing from the available training data is an active area of
machine learning research. However, out-of-distribution generalization for
regression-the analogous problem for modeling continuous targets-remains
relatively unexplored. To tackle this problem, we return to first principles
and analyze how the closed-form solution for Ordinary Least Squares (OLS)
regression is sensitive to covariate shift. We characterize the
out-of-distribution risk of the OLS model in terms of the eigenspectrum
decomposition of the source and target data. We then use this insight to
propose a method for adapting the weights of the last layer of a pre-trained
neural regression model to perform better on input data originating from a
different distribution. We demonstrate how this lightweight spectral adaptation
procedure can improve out-of-distribution performance for synthetic and
real-world datasets.