Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting
Mingkui Tan Guohao Chen Jiaxiang Wu Yifan Zhang Yaofo Chen Peilin Zhao Shuaicheng Niu
Test-time adaptation (TTA) seeks to tackle potential distribution shifts
between training and test data by adapting a given model w.r.t. any test
sample. Although recent TTA has shown promising performance, we still face two
key challenges: 1) prior methods perform backpropagation for each test sample,
resulting in unbearable optimization costs to many applications; 2) while
existing TTA can significantly improve the test performance on
out-of-distribution data, they often suffer from severe performance degradation
on in-distribution data after TTA (known as forgetting). To this end, we have
proposed an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which
develops an active sample selection criterion to identify reliable and
non-redundant samples for test-time entropy minimization. To alleviate
forgetting, EATA introduces a Fisher regularizer estimated from test samples to
constrain important model parameters from drastic changes. However, in EATA,
the adopted entropy loss consistently assigns higher confidence to predictions
even for samples that are underlying uncertain, leading to overconfident
predictions. To tackle this, we further propose EATA with Calibration (EATA-C)
to separately exploit the reducible model uncertainty and the inherent data
uncertainty for calibrated TTA. Specifically, we measure the model uncertainty
by the divergence between predictions from the full network and its
sub-networks, on which we propose a divergence loss to encourage consistent
predictions instead of overconfident ones. To further recalibrate prediction
confidence, we utilize the disagreement among predicted labels as an indicator
of the data uncertainty, and then devise a min-max entropy regularizer to
selectively increase and decrease prediction confidence for different samples.
Experiments on image classification and semantic segmentation verify the
effectiveness of our methods.