Estimating calibration error under label shift without labels
Teodora Popordanoska Gorjan Radevski Tinne Tuytelaars Matthew B. Blaschko
In the face of dataset shift, model calibration plays a pivotal role in
ensuring the reliability of machine learning systems. Calibration error (CE) is
an indicator of the alignment between the predicted probabilities and the
classifier accuracy. While prior works have delved into the implications of
dataset shift on calibration, existing CE estimators assume access to labels
from the target domain, which are often unavailable in practice, i.e., when the
model is deployed and used. This work addresses such challenging scenario, and
proposes a novel CE estimator under label shift, which is characterized by
changes in the marginal label distribution $p(Y)$, while keeping the
conditional $p(X|Y)$ constant between the source and target distributions. Our
contribution is an approach, which, by leveraging importance re-weighting of
the labeled source distribution, provides consistent and asymptotically
unbiased CE estimation with respect to the shifted target distribution.
Empirical results across diverse real-world datasets, under various conditions
and label-shift intensities, demonstrate the effectiveness and reliability of
the proposed estimator.