PseudoCal: A Source-Free Approach to Unsupervised Uncertainty
  Calibration in Domain Adaptation
Dapeng Hu Jian Liang Xinchao Wang Chuan-Sheng Foo
Unsupervised domain adaptation (UDA) has witnessed remarkable advancements in
improving the accuracy of models for unlabeled target domains. However, the
calibration of predictive uncertainty in the target domain, a crucial aspect of
the safe deployment of UDA models, has received limited attention. The
conventional in-domain calibration method, \textit{temperature scaling}
(TempScal), encounters challenges due to domain distribution shifts and the
absence of labeled target domain data. Recent approaches have employed
importance-weighting techniques to estimate the target-optimal temperature
based on re-weighted labeled source data. Nonetheless, these methods require
source data and suffer from unreliable density estimates under severe domain
shifts, rendering them unsuitable for source-free UDA settings. To overcome
these limitations, we propose PseudoCal, a source-free calibration method that
exclusively relies on unlabeled target data. Unlike previous approaches that
treat UDA calibration as a \textit{covariate shift} problem, we consider it as
an unsupervised calibration problem specific to the target domain. Motivated by
the factorization of the negative log-likelihood (NLL) objective in TempScal,
we generate a labeled pseudo-target set that captures the structure of the real
target. By doing so, we transform the unsupervised calibration problem into a
supervised one, enabling us to effectively address it using widely-used
in-domain methods like TempScal. Finally, we thoroughly evaluate the
calibration performance of PseudoCal by conducting extensive experiments on 10
UDA methods, considering both traditional UDA settings and recent source-free
UDA scenarios. The experimental results consistently demonstrate the superior
performance of PseudoCal, exhibiting significantly reduced calibration error
compared to existing calibration methods.