On Pitfalls of Test-Time Adaptation
Hao Zhao Yuejiang Liu Alexandre Alahi Tao Lin
Test-Time Adaptation (TTA) has recently emerged as a promising approach for
tackling the robustness challenge under distribution shifts. However, the lack
of consistent settings and systematic studies in prior literature hinders
thorough assessments of existing methods. To address this issue, we present
TTAB, a test-time adaptation benchmark that encompasses ten state-of-the-art
algorithms, a diverse array of distribution shifts, and two evaluation
protocols. Through extensive experiments, our benchmark reveals three common
pitfalls in prior efforts. First, selecting appropriate hyper-parameters,
especially for model selection, is exceedingly difficult due to online batch
dependency. Second, the effectiveness of TTA varies greatly depending on the
quality and properties of the model being adapted. Third, even under optimal
algorithmic conditions, none of the existing methods are capable of addressing
all common types of distribution shifts. Our findings underscore the need for
future research in the field to conduct rigorous evaluations on a broader set
of models and shifts, and to re-examine the assumptions behind the empirical
success of TTA. Our code is available at
\url{https://github.com/lins-lab/ttab}.