Document Structure in Long Document Transformers
Jan Buchmann Max Eichler Jan-Micha Bodensohn Ilia Kuznetsov Iryna Gurevych
Long documents often exhibit structure with hierarchically organized elements
of different functions, such as section headers and paragraphs. Despite the
omnipresence of document structure, its role in natural language processing
(NLP) remains opaque. Do long-document Transformer models acquire an internal
representation of document structure during pre-training? How can structural
information be communicated to a model after pre-training, and how does it
influence downstream performance? To answer these questions, we develop a novel
suite of probing tasks to assess structure-awareness of long-document
Transformers, propose general-purpose structure infusion methods, and evaluate
the effects of structure infusion on QASPER and Evidence Inference, two
challenging long-document NLP tasks. Results on LED and LongT5 suggest that
they acquire implicit understanding of document structure during pre-training,
which can be further enhanced by structure infusion, leading to improved
end-task performance. To foster research on the role of document structure in
NLP modeling, we make our data and code publicly available.