Transform-Equivariant Consistency Learning for Temporal Sentence
  Grounding
Daizong Liu Xiaoye Qu Jianfeng Dong Pan Zhou Zichuan Xu Haozhao Wang Xing Di Weining Lu Yu Cheng
This paper addresses the temporal sentence grounding (TSG). Although existing
methods have made decent achievements in this task, they not only severely rely
on abundant video-query paired data for training, but also easily fail into the
dataset distribution bias. To alleviate these limitations, we introduce a novel
Equivariant Consistency Regulation Learning (ECRL) framework to learn more
discriminative query-related frame-wise representations for each video, in a
self-supervised manner. Our motivation comes from that the temporal boundary of
the query-guided activity should be consistently predicted under various
video-level transformations. Concretely, we first design a series of
spatio-temporal augmentations on both foreground and background video segments
to generate a set of synthetic video samples. In particular, we devise a
self-refine module to enhance the completeness and smoothness of the augmented
video. Then, we present a novel self-supervised consistency loss (SSCL) applied
on the original and augmented videos to capture their invariant query-related
semantic by minimizing the KL-divergence between the sequence similarity of two
videos and a prior Gaussian distribution of timestamp distance. At last, a
shared grounding head is introduced to predict the transform-equivariant
query-guided segment boundaries for both the original and augmented videos.
Extensive experiments on three challenging datasets (ActivityNet, TACoS, and
Charades-STA) demonstrate both effectiveness and efficiency of our proposed
ECRL framework.