A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension
Weijia Wu Yuzhong Zhao Zhuang Li Jiahong Li Hong Zhou Mike Zheng Shou Xiang Bai
Most existing cross-modal language-to-video retrieval (VR) research focuses
on single-modal input from video, i.e., visual representation, while the text
is omnipresent in human environments and frequently critical to understand
video. To study how to retrieve video with both modal inputs, i.e., visual and
text semantic representations, we first introduce a large-scale and cross-modal
Video Retrieval dataset with text reading comprehension, TextVR, which contains
42.2k sentence queries for 10.5k videos of 8 scenario domains, i.e., Street
View (indoor), Street View (outdoor), Games, Sports, Driving, Activity, TV
Show, and Cooking. The proposed TextVR requires one unified cross-modal model
to recognize and comprehend texts, relate them to the visual context, and
decide what text semantic information is vital for the video retrieval task.
Besides, we present a detailed analysis of TextVR compared to the existing
datasets and design a novel multimodal video retrieval baseline for the
text-based video retrieval task. The dataset analysis and extensive experiments
show that our TextVR benchmark provides many new technical challenges and
insights from previous datasets for the video-and-language community. The
project website and GitHub repo can be found at
https://sites.google.com/view/loveucvpr23/guest-track and
https://github.com/callsys/TextVR, respectively.