Enhancing Visual Document Understanding with Contrastive Learning in
  Large Visual-Language Models
Xin Li Yunfei Wu Xinghua Jiang Zhihao Guo Mingming Gong Haoyu Cao Yinsong Liu Deqiang Jiang Xing Sun
Recently, the advent of Large Visual-Language Models (LVLMs) has received
increasing attention across various domains, particularly in the field of
visual document understanding (VDU). Different from conventional
vision-language tasks, VDU is specifically concerned with text-rich scenarios
containing abundant document elements. Nevertheless, the importance of
fine-grained features remains largely unexplored within the community of LVLMs,
leading to suboptimal performance in text-rich scenarios. In this paper, we
abbreviate it as the fine-grained feature collapse issue. With the aim of
filling this gap, we propose a contrastive learning framework, termed Document
Object COntrastive learning (DoCo), specifically tailored for the downstream
tasks of VDU. DoCo leverages an auxiliary multimodal encoder to obtain the
features of document objects and align them to the visual features generated by
the vision encoder of LVLM, which enhances visual representation in text-rich
scenarios. It can represent that the contrastive learning between the visual
holistic representations and the multimodal fine-grained features of document
objects can assist the vision encoder in acquiring more effective visual cues,
thereby enhancing the comprehension of text-rich documents in LVLMs. We also
demonstrate that the proposed DoCo serves as a plug-and-play pre-training
method, which can be employed in the pre-training of various LVLMs without
inducing any increase in computational complexity during the inference process.
Extensive experimental results on multiple benchmarks of VDU reveal that LVLMs
equipped with our proposed DoCo can achieve superior performance and mitigate
the gap between VDU and generic vision-language tasks.