Softmax Probabilities (Mostly) Predict Large Language Model Correctness
  on Multiple-Choice Q&A
Benjamin Plaut Khanh Nguyen Tu Trinh
Although large language models (LLMs) perform impressively on many tasks,
overconfidence remains a problem. We hypothesized that on multiple-choice Q&A
tasks, wrong answers would be associated with smaller maximum softmax
probabilities (MSPs) compared to correct answers. We comprehensively evaluate
this hypothesis on ten open-source LLMs and five datasets, and find strong
evidence for our hypothesis among models which perform well on the original Q&A
task. For the six LLMs with the best Q&A performance, the AUROC derived from
the MSP was better than random chance with p < 10^{-4} in 59/60 instances.
Among those six LLMs, the average AUROC ranged from 60% to 69%. Leveraging
these findings, we propose a multiple-choice Q&A task with an option to abstain
and show that performance can be improved by selectively abstaining based on
the MSP of the initial model response. We also run the same experiments with
pre-softmax logits instead of softmax probabilities and find similar (but not
identical) results.