From Image to Video, what do we need in multimodal LLMs?
Suyuan Huang Haoxin Zhang Yan Gao Yao Hu Zengchang Qin
Multimodal Large Language Models (MLLMs) have demonstrated profound
capabilities in understanding multimodal information, covering from Image LLMs
to the more complex Video LLMs. Numerous studies have illustrated their
exceptional cross-modal comprehension. Recently, integrating video foundation
models with large language models to build a comprehensive video understanding
system has been proposed to overcome the limitations of specific pre-defined
vision tasks. However, the current advancements in Video LLMs tend to overlook
the foundational contributions of Image LLMs, often opting for more complicated
structures and a wide variety of multimodal data for pre-training. This
approach significantly increases the costs associated with these methods.In
response to these challenges, this work introduces an efficient method that
strategically leverages the priors of Image LLMs, facilitating a
resource-efficient transition from Image to Video LLMs. We propose RED-VILLM, a
Resource-Efficient Development pipeline for Video LLMs from Image LLMs, which
utilizes a temporal adaptation plug-and-play structure within the image fusion
module of Image LLMs. This adaptation extends their understanding capabilities
to include temporal information, enabling the development of Video LLMs that
not only surpass baseline performances but also do so with minimal
instructional data and training resources. Our approach highlights the
potential for a more cost-effective and scalable advancement in multimodal
models, effectively building upon the foundational work of Image LLMs.