On the Need for a Language Describing Distribution Shifts: Illustrations
  on Tabular Datasets
Jiashuo Liu Tianyu Wang Peng Cui Hongseok Namkoong
Different distribution shifts require different algorithmic and operational
interventions. Methodological research must be grounded by the specific shifts
they address. Although nascent benchmarks provide a promising empirical
foundation, they implicitly focus on covariate shifts, and the validity of
empirical findings depends on the type of shift, e.g., previous observations on
algorithmic performance can fail to be valid when the $Y|X$ distribution
changes. We conduct a thorough investigation of natural shifts in 5 tabular
datasets over 86,000 model configurations, and find that $Y|X$-shifts are most
prevalent. To encourage researchers to develop a refined language for
distribution shifts, we build WhyShift, an empirical testbed of curated
real-world shifts where we characterize the type of shift we benchmark
performance over. Since $Y|X$-shifts are prevalent in tabular settings, we
identify covariate regions that suffer the biggest $Y|X$-shifts and discuss
implications for algorithmic and data-based interventions. Our testbed
highlights the importance of future research that builds an understanding of
how distributions differ.