Zero Shot Open-ended Video Inference
Ee Yeo Keat Zhang Hao Alexander Matyasko Basura Fernando
Zero-shot open-ended inference on untrimmed videos poses a significant
challenge, especially when no annotated data is utilized to navigate the
inference direction. In this work, we aim to address this underexplored domain
by introducing an adaptable framework that efficiently combines both the frozen
vision-language (VL) model and off-the-shelf large language model (LLM) for
conducting zero-shot open-ended inference tasks without requiring any
additional training or fine-tuning. Our comprehensive experiments span various
video action datasets for goal inference and action recognition tasks. The
results demonstrate the framework's superior performance in goal inference
compared to conventional vision-language models in open-ended and close-ended
scenarios. Notably, the proposed framework exhibits the capability to
generalize effectively to action recognition tasks, underscoring its
versatility and potential contributions to advancing the video-based zero-shot
understanding.