HowToCaption: Prompting LLMs to Transform Video Annotations at Scale
Nina Shvetsova Anna Kukleva Xudong Hong Christian Rupprecht Bernt Schiele Hilde Kuehne
Instructional videos are an excellent source for learning multimodal
representations by leveraging video-subtitle pairs extracted with automatic
speech recognition systems (ASR) from the audio signal in the videos. However,
in contrast to human-annotated captions, both speech and subtitles naturally
differ from the visual content of the videos and thus provide only noisy
supervision for multimodal learning. As a result, large-scale annotation-free
web video training data remains sub-optimal for training text-video models. In
this work, we propose to leverage the capability of large language models
(LLMs) to obtain fine-grained video descriptions aligned with videos.
Specifically, we prompt an LLM to create plausible video descriptions based on
ASR narrations of the video for a large-scale instructional video dataset. To
this end, we introduce a prompting method that is able to take into account a
longer text of subtitles, allowing us to capture context beyond a single
sentence. To align the captions to the video temporally, we prompt the LLM to
generate timestamps for each produced caption based on the subtitles. In this
way, we obtain human-style video captions at scale without human supervision.
We apply our method to the subtitles of the HowTo100M dataset, creating a new
large-scale dataset, HowToCaption. Our evaluation shows that the resulting
captions not only significantly improve the performance over many different
benchmark datasets for text-video retrieval but also lead to a disentangling of
textual narration from the audio, boosting performance in text-video-audio
tasks.