Classifier Calibration with ROC-Regularized Isotonic Regression
Eugene Berta Francis Bach Michael Jordan
Calibration of machine learning classifiers is necessary to obtain reliable
and interpretable predictions, bridging the gap between model confidence and
actual probabilities. One prominent technique, isotonic regression (IR), aims
at calibrating binary classifiers by minimizing the cross entropy on a
calibration set via monotone transformations. IR acts as an adaptive binning
procedure, which allows achieving a calibration error of zero, but leaves open
the issue of the effect on performance. In this paper, we first prove that IR
preserves the convex hull of the ROC curve -- an essential performance metric
for binary classifiers. This ensures that a classifier is calibrated while
controlling for overfitting of the calibration set. We then present a novel
generalization of isotonic regression to accommodate classifiers with K
classes. Our method constructs a multidimensional adaptive binning scheme on
the probability simplex, again achieving a multi-class calibration error equal
to zero. We regularize this algorithm by imposing a form of monotony that
preserves the K-dimensional ROC surface of the classifier. We show empirically
that this general monotony criterion is effective in striking a balance between
reducing cross entropy loss and avoiding overfitting of the calibration set.