Finding-Aware Anatomical Tokens for Chest X-Ray Automated Reporting
Francesco Dalla Serra Chaoyang Wang Fani Deligianni Jeffrey Dalton Alison Q. O'Neil
The task of radiology reporting comprises describing and interpreting the
medical findings in radiographic images, including description of their
location and appearance. Automated approaches to radiology reporting require
the image to be encoded into a suitable token representation for input to the
language model. Previous methods commonly use convolutional neural networks to
encode an image into a series of image-level feature map representations.
However, the generated reports often exhibit realistic style but imperfect
accuracy. Inspired by recent works for image captioning in the general domain
in which each visual token corresponds to an object detected in an image, we
investigate whether using local tokens corresponding to anatomical structures
can improve the quality of the generated reports. We introduce a novel
adaptation of Faster R-CNN in which finding detection is performed for the
candidate bounding boxes extracted during anatomical structure localisation. We
use the resulting bounding box feature representations as our set of
finding-aware anatomical tokens. This encourages the extracted anatomical
tokens to be informative about the findings they contain (required for the
final task of radiology reporting). Evaluating on the MIMIC-CXR dataset of
chest X-Ray images, we show that task-aware anatomical tokens give
state-of-the-art performance when integrated into an automated reporting
pipeline, yielding generated reports with improved clinical accuracy.