Towards Efficient and Effective Text-to-Video Retrieval with
  Coarse-to-Fine Visual Representation Learning
Kaibin Tian Yanhua Cheng Yi Liu Xinglin Hou Quan Chen Han Li
In recent years, text-to-video retrieval methods based on CLIP have
experienced rapid development. The primary direction of evolution is to exploit
the much wider gamut of visual and textual cues to achieve alignment.
Concretely, those methods with impressive performance often design a heavy
fusion block for sentence (words)-video (frames) interaction, regardless of the
prohibitive computation complexity. Nevertheless, these approaches are not
optimal in terms of feature utilization and retrieval efficiency. To address
this issue, we adopt multi-granularity visual feature learning, ensuring the
model's comprehensiveness in capturing visual content features spanning from
abstract to detailed levels during the training phase. To better leverage the
multi-granularity features, we devise a two-stage retrieval architecture in the
retrieval phase. This solution ingeniously balances the coarse and fine
granularity of retrieval content. Moreover, it also strikes a harmonious
equilibrium between retrieval effectiveness and efficiency. Specifically, in
training phase, we design a parameter-free text-gated interaction block (TIB)
for fine-grained video representation learning and embed an extra Pearson
Constraint to optimize cross-modal representation learning. In retrieval phase,
we use coarse-grained video representations for fast recall of top-k
candidates, which are then reranked by fine-grained video representations.
Extensive experiments on four benchmarks demonstrate the efficiency and
effectiveness. Notably, our method achieves comparable performance with the
current state-of-the-art methods while being nearly 50 times faster.