Open-Set Knowledge-Based Visual Question Answering with Inference Paths
Jingru Gan Xinzhe Han Shuhui Wang Qingming Huang
Given an image and an associated textual question, the purpose of
Knowledge-Based Visual Question Answering (KB-VQA) is to provide a correct
answer to the question with the aid of external knowledge bases. Prior KB-VQA
models are usually formulated as a retriever-classifier framework, where a
pre-trained retriever extracts textual or visual information from knowledge
graphs and then makes a prediction among the candidates. Despite promising
progress, there are two drawbacks with existing models. Firstly, modeling
question-answering as multi-class classification limits the answer space to a
preset corpus and lacks the ability of flexible reasoning. Secondly, the
classifier merely consider "what is the answer" without "how to get the
answer", which cannot ground the answer to explicit reasoning paths. In this
paper, we confront the challenge of \emph{explainable open-set} KB-VQA, where
the system is required to answer questions with entities at wild and retain an
explainable reasoning path. To resolve the aforementioned issues, we propose a
new retriever-ranker paradigm of KB-VQA, Graph pATH rankER (GATHER for
brevity). Specifically, it contains graph constructing, pruning, and path-level
ranking, which not only retrieves accurate answers but also provides inference
paths that explain the reasoning process. To comprehensively evaluate our
model, we reformulate the benchmark dataset OK-VQA with manually corrected
entity-level annotations and release it as ConceptVQA. Extensive experiments on
real-world questions demonstrate that our framework is not only able to perform
open-set question answering across the whole knowledge base but provide
explicit reasoning path.