Prompting Large Language Models to Reformulate Queries for Moment
  Localization
Wenfeng Yan Shaoxiang Chen Zuxuan Wu Yu-Gang Jiang
The task of moment localization is to localize a temporal moment in an
untrimmed video for a given natural language query. Since untrimmed video
contains highly redundant contents, the quality of the query is crucial for
accurately localizing moments, i.e., the query should provide precise
information about the target moment so that the localization model can
understand what to look for in the videos. However, the natural language
queries in current datasets may not be easy to understand for existing models.
For example, the Ego4D dataset uses question sentences as the query to describe
relatively complex moments. While being natural and straightforward for humans,
understanding such question sentences are challenging for mainstream moment
localization models like 2D-TAN. Inspired by the recent success of large
language models, especially their ability of understanding and generating
complex natural language contents, in this extended abstract, we make early
attempts at reformulating the moment queries into a set of instructions using
large language models and making them more friendly to the localization models.