A Challenging Multimodal Video Summary: Simultaneously Extracting and
  Generating Keyframe-Caption Pairs from Video
Keito Kudo Haruki Nagasawa Jun Suzuki Nobuyuki Shimizu
This paper proposes a practical multimodal video summarization task setting
and a dataset to train and evaluate the task. The target task involves
summarizing a given video into a predefined number of keyframe-caption pairs
and displaying them in a listable format to grasp the video content quickly.
This task aims to extract crucial scenes from the video in the form of images
(keyframes) and generate corresponding captions explaining each keyframe's
situation. This task is useful as a practical application and presents a highly
challenging problem worthy of study. Specifically, achieving simultaneous
optimization of the keyframe selection performance and caption quality
necessitates careful consideration of the mutual dependence on both preceding
and subsequent keyframes and captions. To facilitate subsequent research in
this field, we also construct a dataset by expanding upon existing datasets and
propose an evaluation framework. Furthermore, we develop two baseline systems
and report their respective performance.