Modular Visual Question Answering via Code Generation
Sanjay Subramanian Medhini Narasimhan Kushal Khangaonkar Kevin Yang Arsha Nagrani Cordelia Schmid Andy Zeng Trevor Darrell Dan Klein
We present a framework that formulates visual question answering as modular
code generation. In contrast to prior work on modular approaches to VQA, our
approach requires no additional training and relies on pre-trained language
models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA
examples used for in-context learning. The generated Python programs invoke and
compose the outputs of the visual models using arithmetic and conditional
logic. Our approach improves accuracy on the COVR dataset by at least 3% and on
the GQA dataset by roughly 2% compared to the few-shot baseline that does not
employ code generation.