Zero-shot Visual Question Answering with Language Model Feedback
Yifan Du Junyi Li Tianyi Tang Wayne Xin Zhao Ji-Rong Wen
In this paper, we propose a novel language model guided captioning approach,
LAMOC, for knowledge-based visual question answering (VQA). Our approach
employs the generated captions by a captioning model as the context of an
answer prediction model, which is a Pre-trained Language model (PLM). As the
major contribution, we leverage the guidance and feedback of the prediction
model to improve the capability of the captioning model. In this way, the
captioning model can become aware of the task goal and information need from
the PLM. To develop our approach, we design two specific training stages, where
the first stage adapts the captioning model to the prediction model (selecting
more suitable caption propositions for training) and the second stage tunes the
captioning model according to the task goal (learning from feedback of the
PLM). Extensive experiments demonstrate the effectiveness of the proposed
approach on the knowledge-based VQA task. Specifically, on the challenging
A-OKVQA dataset, LAMOC outperforms several competitive zero-shot methods and
even achieves comparable results to a fine-tuned VLP model. Our code is
publicly available at https://github.com/RUCAIBox/LAMOC.