VLAB: Enhancing Video Language Pre-training by Feature Adapting and
  Blending
Xingjian He Sihan Chen Fan Ma Zhicheng Huang Xiaojie Jin Zikang Liu Dongmei Fu Yi Yang Jing Liu Jiashi Feng
Large-scale image-text contrastive pre-training models, such as CLIP, have
been demonstrated to effectively learn high-quality multimodal representations.
However, there is limited research on learning video-text representations for
general video multimodal tasks based on these powerful features. Towards this
goal, we propose a novel video-text pre-training method dubbed VLAB: Video
Language pre-training by feature Adapting and Blending, which transfers CLIP
representations to video pre-training tasks and develops unified video
multimodal models for a wide range of video-text tasks. Specifically, VLAB is
founded on two key strategies: feature adapting and feature blending. In the
former, we introduce a new video adapter module to address CLIP's deficiency in
modeling temporal information and extend the model's capability to encompass
both contrastive and generative tasks. In the latter, we propose an end-to-end
training method that further enhances the model's performance by exploiting the
complementarity of image and video features. We validate the effectiveness and
versatility of VLAB through extensive experiments on highly competitive video
multimodal tasks, including video text retrieval, video captioning, and video
question answering. Remarkably, VLAB outperforms competing methods
significantly and sets new records in video question answering on MSRVTT, MSVD,
and TGIF datasets. It achieves an accuracy of 49.6, 61.0, and 79.0,
respectively. Codes and models will be released.