Selective "Selective Prediction": Reducing Unnecessary Abstention in
  Vision-Language Reasoning
Tejas Srinivasan Jack Hessel Tanmay Gupta Bill Yuchen Lin Yejin Choi Jesse Thomason Khyathi Raghavi Chandu
Prior work on selective prediction minimizes incorrect predictions from
vision-language models (VLMs) by allowing them to abstain from answering when
uncertain. However, when deploying a vision-language system with low tolerance
for inaccurate predictions, selective prediction may be over-cautious and
abstain too frequently, even on many correct predictions. We introduce
ReCoVERR, an inference-time algorithm to reduce the over-abstention of a
selective vision-language system without decreasing prediction accuracy. When
the VLM makes a low-confidence prediction, instead of abstaining ReCoVERR tries
to find relevant clues in the image that provide additional evidence for the
prediction. ReCoVERR uses an LLM to pose related questions to the VLM, collects
high-confidence evidences, and if enough evidence confirms the prediction the
system makes a prediction instead of abstaining. ReCoVERR enables two VLMs,
BLIP2 and InstructBLIP, to answer up to 20% more questions on the A-OKVQA task
than vanilla selective prediction without decreasing system accuracy, thus
improving overall system reliability.