Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video
  Localization
Chongzhi Zhang Mingyuan Zhang Zhiyang Teng Jiayi Li Xizhou Zhu Lewei Lu Ziwei Liu Aixin Sun
Natural Language Video Localization (NLVL), grounding phrases from natural
language descriptions to corresponding video segments, is a complex yet
critical task in video understanding. Despite ongoing advancements, many
existing solutions lack the capability to globally capture temporal dynamics of
the video data. In this study, we present a novel approach to NLVL that aims to
address this issue. Our method involves the direct generation of a global 2D
temporal map via a conditional denoising diffusion process, based on the input
video and language query. The main challenges are the inherent sparsity and
discontinuity of a 2D temporal map in devising the diffusion decoder. To
address these challenges, we introduce a multi-scale technique and develop an
innovative diffusion decoder. Our approach effectively encapsulates the
interaction between the query and video data across various time scales.
Experiments on the Charades and DiDeMo datasets underscore the potency of our
design.