Minimum-Risk Recalibration of Classifiers
Zeyu Sun Dogyoon Song Alfred Hero
Recalibrating probabilistic classifiers is vital for enhancing the
reliability and accuracy of predictive models. Despite the development of
numerous recalibration algorithms, there is still a lack of a comprehensive
theory that integrates calibration and sharpness (which is essential for
maintaining predictive power). In this paper, we introduce the concept of
minimum-risk recalibration within the framework of mean-squared-error (MSE)
decomposition, offering a principled approach for evaluating and recalibrating
probabilistic classifiers. Using this framework, we analyze the uniform-mass
binning (UMB) recalibration method and establish a finite-sample risk upper
bound of order $\tilde{O}(B/n + 1/B^2)$ where $B$ is the number of bins and $n$
is the sample size. By balancing calibration and sharpness, we further
determine that the optimal number of bins for UMB scales with $n^{1/3}$,
resulting in a risk bound of approximately $O(n^{-2/3})$. Additionally, we
tackle the challenge of label shift by proposing a two-stage approach that
adjusts the recalibration function using limited labeled data from the target
domain. Our results show that transferring a calibrated classifier requires
significantly fewer target samples compared to recalibrating from scratch. We
validate our theoretical findings through numerical simulations, which confirm
the tightness of the proposed bounds, the optimal number of bins, and the
effectiveness of label shift adaptation.