Scene Graph Aided Radiology Report Generation
Jun Wang Lixing Zhu Abhir Bhalerao Yulan He
Radiology report generation (RRG) methods often lack sufficient medical
knowledge to produce clinically accurate reports. The scene graph contains rich
information to describe the objects in an image. We explore enriching the
medical knowledge for RRG via a scene graph, which has not been done in the
current RRG literature. To this end, we propose the Scene Graph aided RRG
(SGRRG) network, a framework that generates region-level visual features,
predicts anatomical attributes, and leverages an automatically generated scene
graph, thus achieving medical knowledge distillation in an end-to-end manner.
SGRRG is composed of a dedicated scene graph encoder responsible for
translating the scene graph, and a scene graph-aided decoder that takes
advantage of both patch-level and region-level visual information. A
fine-grained, sentence-level attention method is designed to better dis-till
the scene graph information. Extensive experiments demonstrate that SGRRG
outperforms previous state-of-the-art methods in report generation and can
better capture abnormal findings.