Does Confidence Calibration Help Conformal Prediction?
Huajun Xi Jianguo Huang Lei Feng Hongxin Wei
Conformal prediction, as an emerging uncertainty qualification technique,
constructs prediction sets that are guaranteed to contain the true label with
high probability. Previous works usually employ temperature scaling to
calibrate the classifier, assuming that confidence calibration can benefit
conformal prediction. In this work, we first show that post-hoc calibration
methods surprisingly lead to larger prediction sets with improved calibration,
while over-confidence with small temperatures benefits the conformal prediction
performance instead. Theoretically, we prove that high confidence reduces the
probability of appending a new class in the prediction set. Inspired by the
analysis, we propose a novel method, $\textbf{Conformal Temperature Scaling}$
(ConfTS), which rectifies the objective through the gap between the threshold
and the non-conformity score of the ground-truth label. In this way, the new
objective of ConfTS will optimize the temperature value toward an optimal set
that satisfies the $\textit{marginal coverage}$. Experiments demonstrate that
our method can effectively improve widely-used conformal prediction methods.