Open-vocabulary Video Question Answering: A New Benchmark for Evaluating
  the Generalizability of Video Question Answering Models
Dohwan Ko Ji Soo Lee Miso Choi Jaewon Chu Jihwan Park Hyunwoo J. Kim
Video Question Answering (VideoQA) is a challenging task that entails complex
multi-modal reasoning. In contrast to multiple-choice VideoQA which aims to
predict the answer given several options, the goal of open-ended VideoQA is to
answer questions without restricting candidate answers. However, the majority
of previous VideoQA models formulate open-ended VideoQA as a classification
task to classify the video-question pairs into a fixed answer set, i.e.,
closed-vocabulary, which contains only frequent answers (e.g., top-1000
answers). This leads the model to be biased toward only frequent answers and
fail to generalize on out-of-vocabulary answers. We hence propose a new
benchmark, Open-vocabulary Video Question Answering (OVQA), to measure the
generalizability of VideoQA models by considering rare and unseen answers. In
addition, in order to improve the model's generalization power, we introduce a
novel GNN-based soft verbalizer that enhances the prediction on rare and unseen
answers by aggregating the information from their similar words. For
evaluation, we introduce new baselines by modifying the existing
(closed-vocabulary) open-ended VideoQA models and improve their performances by
further taking into account rare and unseen answers. Our ablation studies and
qualitative analyses demonstrate that our GNN-based soft verbalizer further
improves the model performance, especially on rare and unseen answers. We hope
that our benchmark OVQA can serve as a guide for evaluating the
generalizability of VideoQA models and inspire future research. Code is
available at https://github.com/mlvlab/OVQA.