GenKIE: Robust Generative Multimodal Document Key Information Extraction
Panfeng Cao Ye Wang Qiang Zhang Zaiqiao Meng
Key information extraction (KIE) from scanned documents has gained increasing
attention because of its applications in various domains. Although promising
results have been achieved by some recent KIE approaches, they are usually
built based on discriminative models, which lack the ability to handle optical
character recognition (OCR) errors and require laborious token-level labelling.
In this paper, we propose a novel generative end-to-end model, named GenKIE, to
address the KIE task. GenKIE is a sequence-to-sequence multimodal generative
model that utilizes multimodal encoders to embed visual, layout and textual
features and a decoder to generate the desired output. Well-designed prompts
are leveraged to incorporate the label semantics as the weakly supervised
signals and entice the generation of the key information. One notable advantage
of the generative model is that it enables automatic correction of OCR errors.
Besides, token-level granular annotation is not required. Extensive experiments
on multiple public real-world datasets show that GenKIE effectively generalizes
over different types of documents and achieves state-of-the-art results. Our
experiments also validate the model's robustness against OCR errors, making
GenKIE highly applicable in real-world scenarios.