Domain-adaptive and Subgroup-specific Cascaded Temperature Regression
  for Out-of-distribution Calibration
Jiexin Wang Jiahao Chen Bing Su
Although deep neural networks yield high classification accuracy given
sufficient training data, their predictions are typically overconfident or
under-confident, i.e., the prediction confidences cannot truly reflect the
accuracy. Post-hoc calibration tackles this problem by calibrating the
prediction confidences without re-training the classification model. However,
current approaches assume congruence between test and validation data
distributions, limiting their applicability to out-of-distribution scenarios.
To this end, we propose a novel meta-set-based cascaded temperature regression
method for post-hoc calibration. Our method tailors fine-grained scaling
functions to distinct test sets by simulating various domain shifts through
data augmentation on the validation set. We partition each meta-set into
subgroups based on predicted category and confidence level, capturing diverse
uncertainties. A regression network is then trained to derive category-specific
and confidence-level-specific scaling, achieving calibration across meta-sets.
Extensive experimental results on MNIST, CIFAR-10, and TinyImageNet demonstrate
the effectiveness of the proposed method.