Selective Learning: Towards Robust Calibration with Dynamic
  Regularization
Zongbo Han Yifeng Yang Changqing Zhang Linjun Zhang Joey Tianyi Zhou Qinghua Hu Huaxiu Yao
Miscalibration in deep learning refers to there is a discrepancy between the
predicted confidence and performance. This problem usually arises due to the
overfitting problem, which is characterized by learning everything presented in
the training set, resulting in overconfident predictions during testing.
Existing methods typically address overfitting and mitigate the miscalibration
by adding a maximum-entropy regularizer to the objective function. The
objective can be understood as seeking a model that fits the ground-truth
labels by increasing the confidence while also maximizing the entropy of
predicted probabilities by decreasing the confidence. However, previous methods
lack clear guidance on confidence adjustment, leading to conflicting objectives
(increasing but also decreasing confidence). Therefore, we introduce a method
called Dynamic Regularization (DReg), which aims to learn what should be
learned during training thereby circumventing the confidence adjusting
trade-off. At a high level, DReg aims to obtain a more reliable model capable
of acknowledging what it knows and does not know. Specifically, DReg
effectively fits the labels for in-distribution samples (samples that should be
learned) while applying regularization dynamically to samples beyond model
capabilities (e.g., outliers), thereby obtaining a robust calibrated model
especially on the samples beyond model capabilities. Both theoretical and
empirical analyses sufficiently demonstrate the superiority of DReg compared
with previous methods.