How Flawed is ECE? An Analysis via Logit Smoothing
Muthu Chidambaram Holden Lee Colin McSwiggen Semon Rezchikov
Informally, a model is calibrated if its predictions are correct with a
probability that matches the confidence of the prediction. By far the most
common method in the literature for measuring calibration is the expected
calibration error (ECE). Recent work, however, has pointed out drawbacks of
ECE, such as the fact that it is discontinuous in the space of predictors. In
this work, we ask: how fundamental are these issues, and what are their impacts
on existing results? Towards this end, we completely characterize the
discontinuities of ECE with respect to general probability measures on Polish
spaces. We then use the nature of these discontinuities to motivate a novel
continuous, easily estimated miscalibration metric, which we term
Logit-Smoothed ECE (LS-ECE). By comparing the ECE and LS-ECE of pre-trained
image classification models, we show in initial experiments that binned ECE
closely tracks LS-ECE, indicating that the theoretical pathologies of ECE may
be avoidable in practice.