Self-Consistency Boosts Calibration for Math Reasoning
Ante Wang Linfeng Song Ye Tian Baolin Peng Lifeng Jin Haitao Mi Jinsong Su Dong Yu
Calibration, which establishes the correlation between accuracy and model
confidence, is important for LLM development. We design three off-the-shelf
calibration methods based on self-consistency (Wang et al., 2022) for math
reasoning tasks. Evaluation on two popular benchmarks (GSM8K and MathQA) using
strong open-source LLMs (Mistral and LLaMA2), our methods better bridge model
confidence and accuracy than existing methods based on p(True) (Kadavath et
al., 2022) or logit (Kadavath et al., 2022).