Label Shift Adapter for Test-Time Adaptation under Covariate and Label
  Shifts
Sunghyun Park Seunghan Yang Jaegul Choo Sungrack Yun
Test-time adaptation (TTA) aims to adapt a pre-trained model to the target
domain in a batch-by-batch manner during inference. While label distributions
often exhibit imbalances in real-world scenarios, most previous TTA approaches
typically assume that both source and target domain datasets have balanced
label distribution. Due to the fact that certain classes appear more frequently
in certain domains (e.g., buildings in cities, trees in forests), it is natural
that the label distribution shifts as the domain changes. However, we discover
that the majority of existing TTA methods fail to address the coexistence of
covariate and label shifts. To tackle this challenge, we propose a novel label
shift adapter that can be incorporated into existing TTA approaches to deal
with label shifts during the TTA process effectively. Specifically, we estimate
the label distribution of the target domain to feed it into the label shift
adapter. Subsequently, the label shift adapter produces optimal parameters for
the target label distribution. By predicting only the parameters for a part of
the pre-trained source model, our approach is computationally efficient and can
be easily applied, regardless of the model architectures. Through extensive
experiments, we demonstrate that integrating our strategy with TTA approaches
leads to substantial performance improvements under the joint presence of label
and covariate shifts.