Properties of the ENCE and other MAD-based calibration metrics
Pascal Pernot
The Expected Normalized Calibration Error (ENCE) is a popular calibration
statistic used in Machine Learning to assess the quality of prediction
uncertainties for regression problems. Estimation of the ENCE is based on the
binning of calibration data. In this short note, I illustrate an annoying
property of the ENCE, i.e. its proportionality to the square root of the number
of bins for well calibrated or nearly calibrated datasets. A similar behavior
affects the calibration error based on the variance of z-scores (ZVE), and in
both cases this property is a consequence of the use of a Mean Absolute
Deviation (MAD) statistic to estimate calibration errors. Hence, the question
arises of which number of bins to choose for a reliable estimation of
calibration error statistics. A solution is proposed to infer ENCE and ZVE
values that do not depend on the number of bins for datasets assumed to be
calibrated, providing simultaneously a statistical calibration test. It is also
shown that the ZVE is less sensitive than the ENCE to outstanding errors or
uncertainties.