Consistent and Asymptotically Unbiased Estimation of Proper Calibration
  Errors
Teodora Popordanoska Sebastian G. Gruber Aleksei Tiulpin Florian Buettner Matthew B. Blaschko
Proper scoring rules evaluate the quality of probabilistic predictions,
playing an essential role in the pursuit of accurate and well-calibrated
models. Every proper score decomposes into two fundamental components -- proper
calibration error and refinement -- utilizing a Bregman divergence. While
uncertainty calibration has gained significant attention, current literature
lacks a general estimator for these quantities with known statistical
properties. To address this gap, we propose a method that allows consistent,
and asymptotically unbiased estimation of all proper calibration errors and
refinement terms. In particular, we introduce Kullback--Leibler calibration
error, induced by the commonly used cross-entropy loss. As part of our results,
we prove the relation between refinement and f-divergences, which implies
information monotonicity in neural networks, regardless of which proper scoring
rule is optimized. Our experiments validate empirically the claimed properties
of the proposed estimator and suggest that the selection of a post-hoc
calibration method should be determined by the particular calibration error of
interest.