Visual Grounding of Whole Radiology Reports for 3D CT Images
Akimichi Ichinose Taro Hatsutani Keigo Nakamura Yoshiro Kitamura Satoshi Iizuka Edgar Simo-Serra Shoji Kido Noriyuki Tomiyama
Building a large-scale training dataset is an essential problem in the
development of medical image recognition systems. Visual grounding techniques,
which automatically associate objects in images with corresponding
descriptions, can facilitate labeling of large number of images. However,
visual grounding of radiology reports for CT images remains challenging,
because so many kinds of anomalies are detectable via CT imaging, and resulting
report descriptions are long and complex. In this paper, we present the first
visual grounding framework designed for CT image and report pairs covering
various body parts and diverse anomaly types. Our framework combines two
components of 1) anatomical segmentation of images, and 2) report structuring.
The anatomical segmentation provides multiple organ masks of given CT images,
and helps the grounding model recognize detailed anatomies. The report
structuring helps to accurately extract information regarding the presence,
location, and type of each anomaly described in corresponding reports. Given
the two additional image/report features, the grounding model can achieve
better localization. In the verification process, we constructed a large-scale
dataset with region-description correspondence annotations for 10,410 studies
of 7,321 unique patients. We evaluated our framework using grounding accuracy,
the percentage of correctly localized anomalies, as a metric and demonstrated
that the combination of the anatomical segmentation and the report structuring
improves the performance with a large margin over the baseline model (66.0% vs
77.8%). Comparison with the prior techniques also showed higher performance of
our method.