Dual Focal Loss for Calibration
Linwei Tao Minjing Dong Chang Xu
The use of deep neural networks in real-world applications require
well-calibrated networks with confidence scores that accurately reflect the
actual probability. However, it has been found that these networks often
provide over-confident predictions, which leads to poor calibration. Recent
efforts have sought to address this issue by focal loss to reduce
over-confidence, but this approach can also lead to under-confident
predictions. While different variants of focal loss have been explored, it is
difficult to find a balance between over-confidence and under-confidence. In
our work, we propose a new loss function by focusing on dual logits. Our method
not only considers the ground truth logit, but also take into account the
highest logit ranked after the ground truth logit. By maximizing the gap
between these two logits, our proposed dual focal loss can achieve a better
balance between over-confidence and under-confidence. We provide theoretical
evidence to support our approach and demonstrate its effectiveness through
evaluations on multiple models and datasets, where it achieves state-of-the-art
performance. Code is available at https://github.com/Linwei94/DualFocalLoss