Fusing Multimodal Signals on Hyper-complex Space for Extreme Abstractive
  Text Summarization (TL;DR) of Scientific Contents
Yash Kumar Atri Vikram Goyal Tanmoy Chakraborty
The realm of scientific text summarization has experienced remarkable
progress due to the availability of annotated brief summaries and ample data.
However, the utilization of multiple input modalities, such as videos and
audio, has yet to be thoroughly explored. At present, scientific
multimodal-input-based text summarization systems tend to employ longer target
summaries like abstracts, leading to an underwhelming performance in the task
of text summarization.
  In this paper, we deal with a novel task of extreme abstractive text
summarization (aka TL;DR generation) by leveraging multiple input modalities.
To this end, we introduce mTLDR, a first-of-its-kind dataset for the
aforementioned task, comprising videos, audio, and text, along with both
author-composed summaries and expert-annotated summaries. The mTLDR dataset
accompanies a total of 4,182 instances collected from various academic
conference proceedings, such as ICLR, ACL, and CVPR. Subsequently, we present
mTLDRgen, an encoder-decoder-based model that employs a novel dual-fused
hyper-complex Transformer combined with a Wasserstein Riemannian Encoder
Transformer, to dexterously capture the intricacies between different
modalities in a hyper-complex latent geometric space. The hyper-complex
Transformer captures the intrinsic properties between the modalities, while the
Wasserstein Riemannian Encoder Transformer captures the latent structure of the
modalities in the latent space geometry, thereby enabling the model to produce
diverse sentences. mTLDRgen outperforms 20 baselines on mTLDR as well as
another non-scientific dataset (How2) across three Rouge-based evaluation
measures. Furthermore, based on the qualitative metrics, BERTScore and FEQA,
and human evaluations, we demonstrate that the summaries generated by mTLDRgen
are fluent and congruent to the original source material.