TransferDoc: A Self-Supervised Transferable Document Representation
  Learning Model Unifying Vision and Language
Souhail Bakkali Sanket Biswas Zuheng Ming Mickael Coustaty Marçal Rusiñol Oriol Ramos Terrades Josep Lladós
The field of visual document understanding has witnessed a rapid growth in
emerging challenges and powerful multi-modal strategies. However, they rely on
an extensive amount of document data to learn their pretext objectives in a
``pre-train-then-fine-tune'' paradigm and thus, suffer a significant
performance drop in real-world online industrial settings. One major reason is
the over-reliance on OCR engines to extract local positional information within
a document page. Therefore, this hinders the model's generalizability,
flexibility and robustness due to the lack of capturing global information
within a document image. We introduce TransferDoc, a cross-modal
transformer-based architecture pre-trained in a self-supervised fashion using
three novel pretext objectives. TransferDoc learns richer semantic concepts by
unifying language and visual representations, which enables the production of
more transferable models. Besides, two novel downstream tasks have been
introduced for a ``closer-to-real'' industrial evaluation scenario where
TransferDoc outperforms other state-of-the-art approaches.