VQA-GEN: A Visual Question Answering Benchmark for Domain Generalization
Suraj Jyothi Unni Raha Moraffah Huan Liu
Visual question answering (VQA) models are designed to demonstrate
visual-textual reasoning capabilities. However, their real-world applicability
is hindered by a lack of comprehensive benchmark datasets. Existing domain
generalization datasets for VQA exhibit a unilateral focus on textual shifts
while VQA being a multi-modal task contains shifts across both visual and
textual domains. We propose VQA-GEN, the first ever multi-modal benchmark
dataset for distribution shift generated through a shift induced pipeline.
Experiments demonstrate VQA-GEN dataset exposes the vulnerability of existing
methods to joint multi-modal distribution shifts. validating that comprehensive
multi-modal shifts are critical for robust VQA generalization. Models trained
on VQA-GEN exhibit improved cross-domain and in-domain performance, confirming
the value of VQA-GEN. Further, we analyze the importance of each shift
technique of our pipeline contributing to the generalization of the model.