Scaling Up Video Summarization Pretraining with Large Language Models
Dawit Mureja Argaw Seunghyun Yoon Fabian Caba Heilbron Hanieh Deilamsalehy Trung Bui Zhaowen Wang Franck Dernoncourt Joon Son Chung
Long-form video content constitutes a significant portion of internet
traffic, making automated video summarization an essential research problem.
However, existing video summarization datasets are notably limited in their
size, constraining the effectiveness of state-of-the-art methods for
generalization. Our work aims to overcome this limitation by capitalizing on
the abundance of long-form videos with dense speech-to-video alignment and the
remarkable capabilities of recent large language models (LLMs) in summarizing
long text. We introduce an automated and scalable pipeline for generating a
large-scale video summarization dataset using LLMs as Oracle summarizers. By
leveraging the generated dataset, we analyze the limitations of existing
approaches and propose a new video summarization model that effectively
addresses them. To facilitate further research in the field, our work also
presents a new benchmark dataset that contains 1200 long videos each with
high-quality summaries annotated by professionals. Extensive experiments
clearly indicate that our proposed approach sets a new state-of-the-art in
video summarization across several benchmarks.