Zero-Shot Video Question Answering with Procedural Programs
Rohan Choudhury Koichiro Niinuma Kris M. Kitani László A. Jeni
We propose to answer zero-shot questions about videos by generating short
procedural programs that derive a final answer from solving a sequence of
visual subtasks. We present Procedural Video Querying (ProViQ), which uses a
large language model to generate such programs from an input question and an
API of visual modules in the prompt, then executes them to obtain the output.
Recent similar procedural approaches have proven successful for image question
answering, but videos remain challenging: we provide ProViQ with modules
intended for video understanding, allowing it to generalize to a wide variety
of videos. This code generation framework additionally enables ProViQ to
perform other video tasks in addition to question answering, such as
multi-object tracking or basic video editing. ProViQ achieves state-of-the-art
results on a diverse range of benchmarks, with improvements of up to 25% on
short, long, open-ended, and multimodal video question-answering datasets. Our
project page is at https://rccchoudhury.github.io/proviq2023.