Probabilistic Calibration by Design for Neural Network Regression
Victor Dheur Souhaib Ben Taieb
Generating calibrated and sharp neural network predictive distributions for
regression problems is essential for optimal decision-making in many real-world
applications. To address the miscalibration issue of neural networks, various
methods have been proposed to improve calibration, including post-hoc methods
that adjust predictions after training and regularization methods that act
during training. While post-hoc methods have shown better improvement in
calibration compared to regularization methods, the post-hoc step is completely
independent of model training. We introduce a novel end-to-end model training
procedure called Quantile Recalibration Training, integrating post-hoc
calibration directly into the training process without additional parameters.
We also present a unified algorithm that includes our method and other post-hoc
and regularization methods, as particular cases. We demonstrate the performance
of our method in a large-scale experiment involving 57 tabular regression
datasets, showcasing improved predictive accuracy while maintaining
calibration. We also conduct an ablation study to evaluate the significance of
different components within our proposed method, as well as an in-depth
analysis of the impact of the base model and different hyperparameters on
predictive accuracy.