Intensive Vision-guided Network for Radiology Report Generation
Fudan Zheng Mengfei Li Ying Wang Weijiang Yu Ruixuan Wang Zhiguang Chen Nong Xiao Yutong Lu
Automatic radiology report generation is booming due to its huge application
potential for the healthcare industry. However, existing computer vision and
natural language processing approaches to tackle this problem are limited in
two aspects. First, when extracting image features, most of them neglect
multi-view reasoning in vision and model single-view structure of medical
images, such as space-view or channel-view. However, clinicians rely on
multi-view imaging information for comprehensive judgment in daily clinical
diagnosis. Second, when generating reports, they overlook context reasoning
with multi-modal information and focus on pure textual optimization utilizing
retrieval-based methods. We aim to address these two issues by proposing a
model that better simulates clinicians' perspectives and generates more
accurate reports. Given the above limitation in feature extraction, we propose
a Globally-intensive Attention (GIA) module in the medical image encoder to
simulate and integrate multi-view vision perception. GIA aims to learn three
types of vision perception: depth view, space view, and pixel view. On the
other hand, to address the above problem in report generation, we explore how
to involve multi-modal signals to generate precisely matched reports, i.e., how
to integrate previously predicted words with region-aware visual content in
next word prediction. Specifically, we design a Visual Knowledge-guided Decoder
(VKGD), which can adaptively consider how much the model needs to rely on
visual information and previously predicted text to assist next word
prediction. Hence, our final Intensive Vision-guided Network (IVGN) framework
includes a GIA-guided Visual Encoder and the VKGD. Experiments on two
commonly-used datasets IU X-Ray and MIMIC-CXR demonstrate the superior ability
of our method compared with other state-of-the-art approaches.