Language Guided Visual Question Answering: Elevate Your Multimodal
  Language Model Using Knowledge-Enriched Prompts
Deepanway Ghosal Navonil Majumder Roy Ka-Wei Lee Rada Mihalcea Soujanya Poria
Visual question answering (VQA) is the task of answering questions about an
image. The task assumes an understanding of both the image and the question to
provide a natural language answer. VQA has gained popularity in recent years
due to its potential applications in a wide range of fields, including
robotics, education, and healthcare. In this paper, we focus on
knowledge-augmented VQA, where answering the question requires commonsense
knowledge, world knowledge, and reasoning about ideas and concepts not present
in the image. We propose a multimodal framework that uses language guidance
(LG) in the form of rationales, image captions, scene graphs, etc to answer
questions more accurately. We benchmark our method on the multi-choice
question-answering task of the A-OKVQA, Science-QA, VSR, and IconQA datasets
using CLIP and BLIP models. We show that the use of language guidance is a
simple but powerful and effective strategy for visual question answering. Our
language guidance improves the performance of CLIP by 7.6% and BLIP-2 by 4.8%
in the challenging A-OKVQA dataset. We also observe consistent improvement in
performance on the Science-QA, VSR, and IconQA datasets when using the proposed
language guidances. The implementation of LG-VQA is publicly available at
https:// github.com/declare-lab/LG-VQA.