Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided
  Dynamic Token Merge for Document Understanding
Mingliang Zhai Yulin Li Xiameng Qin Chen Yi Qunyi Xie Chengquan Zhang Kun Yao Yuwei Wu Yunde Jia
Transformers achieve promising performance in document understanding because
of their high effectiveness and still suffer from quadratic computational
complexity dependency on the sequence length. General efficient transformers
are challenging to be directly adapted to model document. They are unable to
handle the layout representation in documents, e.g. word, line and paragraph,
on different granularity levels and seem hard to achieve a good trade-off
between efficiency and performance. To tackle the concerns, we propose
Fast-StrucTexT, an efficient multi-modal framework based on the StrucTexT
algorithm with an hourglass transformer architecture, for visual document
understanding. Specifically, we design a modality-guided dynamic token merging
block to make the model learn multi-granularity representation and prunes
redundant tokens. Additionally, we present a multi-modal interaction module
called Symmetry Cross Attention (SCA) to consider multi-modal fusion and
efficiently guide the token mergence. The SCA allows one modality input as
query to calculate cross attention with another modality in a dual phase.
Extensive experiments on FUNSD, SROIE, and CORD datasets demonstrate that our
model achieves the state-of-the-art performance and almost 1.9X faster
inference time than the state-of-the-art methods.