Language Repository for Long Video Understanding
Kumara Kahatapitiya Kanchana Ranasinghe Jongwoo Park Michael S. Ryoo
Language has become a prominent modality in computer vision with the rise of
multi-modal LLMs. Despite supporting long context-lengths, their effectiveness
in handling long-term information gradually declines with input length. This
becomes critical, especially in applications such as long-form video
understanding. In this paper, we introduce a Language Repository (LangRepo) for
LLMs, that maintains concise and structured information as an interpretable
(i.e., all-textual) representation. Our repository is updated iteratively based
on multi-scale video chunks. We introduce write and read operations that focus
on pruning redundancies in text, and extracting information at various temporal
scales. The proposed framework is evaluated on zero-shot visual
question-answering benchmarks including EgoSchema, NExT-QA, IntentQA and
NExT-GQA, showing state-of-the-art performance at its scale. Our code is
available at https://github.com/kkahatapitiya/LangRepo.