Joint Moment Retrieval and Highlight Detection Via Natural Language
  Queries
Richard Luo Austin Peng Heidi Yap Koby Beard
Video summarization has become an increasingly important task in the field of
computer vision due to the vast amount of video content available on the
internet. In this project, we propose a new method for natural language query
based joint video summarization and highlight detection using multi-modal
transformers. This approach will use both visual and audio cues to match a
user's natural language query to retrieve the most relevant and interesting
moments from a video. Our approach employs multiple recent techniques used in
Vision Transformers (ViTs) to create a transformer-like encoder-decoder model.
We evaluated our approach on multiple datasets such as YouTube Highlights and
TVSum to demonstrate the flexibility of our proposed method.