Can Model Fusing Help Transformers in Long Document Classification? An
  Empirical Study
Damith Premasiri Tharindu Ranasinghe Ruslan Mitkov
Text classification is an area of research which has been studied over the
years in Natural Language Processing (NLP). Adapting NLP to multiple domains
has introduced many new challenges for text classification and one of them is
long document classification. While state-of-the-art transformer models provide
excellent results in text classification, most of them have limitations in the
maximum sequence length of the input sequence. The majority of the transformer
models are limited to 512 tokens, and therefore, they struggle with long
document classification problems. In this research, we explore on employing
Model Fusing for long document classification while comparing the results with
well-known BERT and Longformer architectures.