Is this model reliable for everyone? Testing for strong calibration
Jean Feng Alexej Gossmann Romain Pirracchio Nicholas Petrick Gene Pennello Berkman Sahiner
In a well-calibrated risk prediction model, the average predicted probability
is close to the true event rate for any given subgroup. Such models are
reliable across heterogeneous populations and satisfy strong notions of
algorithmic fairness. However, the task of auditing a model for strong
calibration is well-known to be difficult -- particularly for machine learning
(ML) algorithms -- due to the sheer number of potential subgroups. As such,
common practice is to only assess calibration with respect to a few predefined
subgroups. Recent developments in goodness-of-fit testing offer potential
solutions but are not designed for settings with weak signal or where the
poorly calibrated subgroup is small, as they either overly subdivide the data
or fail to divide the data at all. We introduce a new testing procedure based
on the following insight: if we can reorder observations by their expected
residuals, there should be a change in the association between the predicted
and observed residuals along this sequence if a poorly calibrated subgroup
exists. This lets us reframe the problem of calibration testing into one of
changepoint detection, for which powerful methods already exist. We begin with
introducing a sample-splitting procedure where a portion of the data is used to
train a suite of candidate models for predicting the residual, and the
remaining data are used to perform a score-based cumulative sum (CUSUM) test.
To further improve power, we then extend this adaptive CUSUM test to
incorporate cross-validation, while maintaining Type I error control under
minimal assumptions. Compared to existing methods, the proposed procedure
consistently achieved higher power in simulation studies and more than doubled
the power when auditing a mortality risk prediction model.