Reducing Hallucinations: Enhancing VQA for Flood Disaster Damage
  Assessment with Visual Contexts
Yimin Sun Chao Wang Yan Peng
The zero-shot performance of visual question answering (VQA) models relies
heavily on prompts. For example, a zero-shot VQA for disaster scenarios could
leverage well-designed Chain of Thought (CoT) prompts to stimulate the model's
potential. However, using CoT prompts has some problems, such as causing an
incorrect answer in the end due to the hallucination in the thought process. In
this paper, we propose a zero-shot VQA named Flood Disaster VQA with Two-Stage
Prompt (VQA-TSP). The model generates the thought process in the first stage
and then uses the thought process to generate the final answer in the second
stage. In particular, visual context is added in the second stage to relieve
the hallucination problem that exists in the thought process. Experimental
results show that our method exceeds the performance of state-of-the-art
zero-shot VQA models for flood disaster scenarios in total. Our study provides
a research basis for improving the performance of CoT-based zero-shot VQA.