ViGT: Proposal-free Video Grounding with Learnable Token in Transformer
Kun Li Dan Guo Meng Wang
The video grounding (VG) task aims to locate the queried action or event in
an untrimmed video based on rich linguistic descriptions. Existing
proposal-free methods are trapped in complex interaction between video and
query, overemphasizing cross-modal feature fusion and feature correlation for
VG. In this paper, we propose a novel boundary regression paradigm that
performs regression token learning in a transformer. Particularly, we present a
simple but effective proposal-free framework, namely Video Grounding
Transformer (ViGT), which predicts the temporal boundary using a learnable
regression token rather than multi-modal or cross-modal features. In ViGT, the
benefits of a learnable token are manifested as follows. (1) The token is
unrelated to the video or the query and avoids data bias toward the original
video and query. (2) The token simultaneously performs global context
aggregation from video and query features. First, we employed a sharing feature
encoder to project both video and query into a joint feature space before
performing cross-modal co-attention (i.e., video-to-query attention and
query-to-video attention) to highlight discriminative features in each
modality. Furthermore, we concatenated a learnable regression token [REG] with
the video and query features as the input of a vision-language transformer.
Finally, we utilized the token [REG] to predict the target moment and visual
features to constrain the foreground and background probabilities at each
timestamp. The proposed ViGT performed well on three public datasets: ANet
Captions, TACoS and YouCookII. Extensive ablation studies and qualitative
analysis further validated the interpretability of ViGT.