KNVQA: A Benchmark for evaluation knowledge-based VQA
Sirui Cheng Siyu Zhang Jiayi Wu Muchen Lan
Within the multimodal field, large vision-language models (LVLMs) have made
significant progress due to their strong perception and reasoning capabilities
in the visual and language systems. However, LVLMs are still plagued by the two
critical issues of object hallucination and factual accuracy, which limit the
practicality of LVLMs in different scenarios. Furthermore, previous evaluation
methods focus more on the comprehension and reasoning of language content but
lack a comprehensive evaluation of multimodal interactions, thereby resulting
in potential limitations. To this end, we propose a novel KNVQA-Eval, which is
devoted to knowledge-based VQA task evaluation to reflect the factuality of
multimodal LVLMs. To ensure the robustness and scalability of the evaluation,
we develop a new KNVQA dataset by incorporating human judgment and perception,
aiming to evaluate the accuracy of standard answers relative to AI-generated
answers in knowledge-based VQA. This work not only comprehensively evaluates
the contextual information of LVLMs using reliable human annotations, but also
further analyzes the fine-grained capabilities of current methods to reveal
potential avenues for subsequent optimization of LVLMs-based estimators. Our
proposed VQA-Eval and corresponding dataset KNVQA will facilitate the
development of automatic evaluation tools with the advantages of low cost,
privacy protection, and reproducibility. Our code will be released upon
publication.