Investigating Text Shortening Strategy in BERT: Truncation vs
  Summarization
Mirza Alim Mutasodirin Radityo Eko Prasojo
The parallelism of Transformer-based models comes at the cost of their input
max-length. Some studies proposed methods to overcome this limitation, but none
of them reported the effectiveness of summarization as an alternative. In this
study, we investigate the performance of document truncation and summarization
in text classification tasks. Each of the two was investigated with several
variations. This study also investigated how close their performances are to
the performance of full-text. We used a dataset of summarization tasks based on
Indonesian news articles (IndoSum) to do classification tests. This study shows
how the summaries outperform the majority of truncation method variations and
lose to only one. The best strategy obtained in this study is taking the head
of the document. The second is extractive summarization. This study explains
what happened to the result, leading to further research in order to exploit
the potential of document summarization as a shortening alternative. The code
and data used in this work are publicly available in
https://github.com/mirzaalimm/TruncationVsSummarization.