Benchmarking Test-Time Adaptation against Distribution Shifts in Image
  Classification
Yongcan Yu Lijun Sheng Ran He Jian Liang
Test-time adaptation (TTA) is a technique aimed at enhancing the
generalization performance of models by leveraging unlabeled samples solely
during prediction. Given the need for robustness in neural network systems when
faced with distribution shifts, numerous TTA methods have recently been
proposed. However, evaluating these methods is often done under different
settings, such as varying distribution shifts, backbones, and designing
scenarios, leading to a lack of consistent and fair benchmarks to validate
their effectiveness. To address this issue, we present a benchmark that
systematically evaluates 13 prominent TTA methods and their variants on five
widely used image classification datasets: CIFAR-10-C, CIFAR-100-C, ImageNet-C,
DomainNet, and Office-Home. These methods encompass a wide range of adaptation
scenarios (e.g. online adaptation v.s. offline adaptation, instance adaptation
v.s. batch adaptation v.s. domain adaptation). Furthermore, we explore the
compatibility of different TTA methods with diverse network backbones. To
implement this benchmark, we have developed a unified framework in PyTorch,
which allows for consistent evaluation and comparison of the TTA methods across
the different datasets and network architectures. By establishing this
benchmark, we aim to provide researchers and practitioners with a reliable
means of assessing and comparing the effectiveness of TTA methods in improving
model robustness and generalization performance. Our code is available at
https://github.com/yuyongcan/Benchmark-TTA.