Learning Summary-Worthy Visual Representation for Abstractive
  Summarization in Video
Zenan Xu Xiaojun Meng Yasheng Wang Qinliang Su Zexuan Qiu Xin Jiang Qun Liu
Multimodal abstractive summarization for videos (MAS) requires generating a
concise textual summary to describe the highlights of a video according to
multimodal resources, in our case, the video content and its transcript.
Inspired by the success of the large-scale generative pre-trained language
model (GPLM) in generating high-quality textual content (e.g., summary), recent
MAS methods have proposed to adapt the GPLM to this task by equipping it with
the visual information, which is often obtained through a general-purpose
visual feature extractor. However, the generally extracted visual features may
overlook some summary-worthy visual information, which impedes model
performance. In this work, we propose a novel approach to learning the
summary-worthy visual representation that facilitates abstractive
summarization. Our method exploits the summary-worthy information from both the
cross-modal transcript data and the knowledge that distills from the pseudo
summary. Extensive experiments on three public multimodal datasets show that
our method outperforms all competing baselines. Furthermore, with the
advantages of summary-worthy visual information, our model can have a
significant improvement on small datasets or even datasets with limited
training data.