Improving Video Corpus Moment Retrieval with Partial Relevance
  Enhancement
Danyang Hou Liang Pang Huawei Shen Xueqi Cheng
Video corpus moment retrieval~(VCMR) is a new video retrieval task aimed at
retrieving a relevant moment from a large corpus of untrimmed videos using a
natural language text as query. The relevance between the video and query is
partial, mainly evident in two aspects: (1) Scope: The untrimmed video contains
information-rich frames, and not all are relevant to the query. Strong
correlation is typically observed only within the relevant moment, emphasizing
the importance of capturing key content. (2) Modality: The relevance of query
to different modalities varies; action descriptions align more with the visual
elements, while character conversations are more related to textual
information. Recognizing and addressing these modality-specific nuances is
crucial for effective retrieval in VCMR. However, existing methods often treat
all video contents equally, leading to sub-optimal moment retrieval. We argue
that effectively capturing the partial relevance between the query and video is
essential for the VCMR task. To this end, we propose a Partial Relevance
Enhanced Model~(PREM) to improve VCMR. VCMR involves two sub-tasks: video
retrieval and moment localization. To align with their distinct objectives, we
implement specialized partial relevance enhancement strategies. For video
retrieval, we introduce a multi-modal collaborative video retriever, generating
distinct query representations tailored for different modalities by
modality-specific pooling, ensuring a more effective match. For moment
localization, we propose the focus-then-fuse moment localizer, utilizing
modality-specific gates to capture essential content, followed by fusing
multi-modal information for moment localization. Experimental results on TVR
and DiDeMo datasets show that the proposed model outperforms the baselines,
achieving a new state-of-the-art of VCMR.