Scene Summarization: Clustering Scene Videos into Spatially Diverse
  Frames
Chao Chen Mingzhi Zhu Ankush Pratap Singh Yu Yan Felix Juefei Xu Chen Feng
We propose scene summarization as a new video-based scene understanding task.
It aims to summarize a long video walkthrough of a scene into a small set of
frames that are spatially diverse in the scene, which has many impotant
applications, such as in surveillance, real estate, and robotics. It stems from
video summarization but focuses on long and continuous videos from moving
cameras, instead of user-edited fragmented video clips that are more commonly
studied in existing video summarization works. Our solution to this task is a
two-stage self-supervised pipeline named SceneSum. Its first stage uses
clustering to segment the video sequence. Our key idea is to combine visual
place recognition (VPR) into this clustering process to promote spatial
diversity. Its second stage needs to select a representative keyframe from each
cluster as the summary while respecting resource constraints such as memory and
disk space limits. Additionally, if the ground truth image trajectory is
available, our method can be easily augmented with a supervised loss to enhance
the clustering and keyframe selection. Extensive experiments on both real-world
and simulated datasets show our method outperforms common video summarization
baselines by 50%