Conditional Modeling Based Automatic Video Summarization
Jia-Hong Huang Chao-Han Huck Yang Pin-Yu Chen Min-Hung Chen Marcel Worring
The aim of video summarization is to shorten videos automatically while
retaining the key information necessary to convey the overall story. Video
summarization methods mainly rely on visual factors, such as visual
consecutiveness and diversity, which may not be sufficient to fully understand
the content of the video. There are other non-visual factors, such as
interestingness, representativeness, and storyline consistency that should also
be considered for generating high-quality video summaries. Current methods do
not adequately take into account these non-visual factors, resulting in
suboptimal performance. In this work, a new approach to video summarization is
proposed based on insights gained from how humans create ground truth video
summaries. The method utilizes a conditional modeling perspective and
introduces multiple meaningful random variables and joint distributions to
characterize the key components of video summarization. Helper distributions
are employed to improve the training of the model. A conditional attention
module is designed to mitigate potential performance degradation in the
presence of multi-modal input. The proposed video summarization method
incorporates the above innovative design choices that aim to narrow the gap
between human-generated and machine-generated video summaries. Extensive
experiments show that the proposed approach outperforms existing methods and
achieves state-of-the-art performance on commonly used video summarization
datasets.