S4M: Generating Radiology Reports by A Single Model for Multiple Body
  Parts
Qi Chen Yutong Xie Biao Wu Minh-Son To James Ang Qi Wu
In this paper, we seek to design a report generation model that is able to
generate reasonable reports even given different images of various body parts.
We start by directly merging multiple datasets and training a single report
generation model on this one. We, however, observe that the reports generated
in such a simple way only obtain comparable performance compared with that
trained separately on each specific dataset. We suspect that this is caused by
the dilemma between the diversity of body parts and the limited availability of
medical data. To develop robust and generalizable models, it is important to
consider a diverse range of body parts and medical conditions. However,
collecting a sufficiently large dataset for each specific body part can be
difficult due to various factors, such as data availability and privacy
concerns. Thus, rather than striving for more data, we propose a
single-for-multiple (S4M) framework, which seeks to facilitate the learning of
the report generation model with two auxiliary priors: an explicit prior (\ie,
feeding radiology-informed knowledge) and an implicit prior (\ie, guided by
cross-modal features). Specifically, based on the conventional encoder-decoder
report generation framework, we incorporate two extra branches: a
Radiology-informed Knowledge Aggregation (RadKA) branch and an Implicit Prior
Guidance (IPG) branch. We conduct the experiments on our merged dataset which
consists of a public dataset (\ie, IU-Xray) and five private datasets, covering
six body parts: chest, abdomen, knee, hip, wrist and shoulder. Our S4M model
outperforms all the baselines, regardless of whether they are trained on
separate or merged datasets. Code is available at:
\url{https://github.com/YtongXie/S4M}.