C^2M-DoT: Cross-modal consistent multi-view medical report generation
  with domain transfer network
Ruizhi Wang Xiangtao Wang Jie Zhou Thomas Lukasiewicz Zhenghua Xu
In clinical scenarios, multiple medical images with different views are
usually generated simultaneously, and these images have high semantic
consistency. However, most existing medical report generation methods only
consider single-view data. The rich multi-view mutual information of medical
images can help generate more accurate reports, however, the dependence of
multi-view models on multi-view data in the inference stage severely limits
their application in clinical practice. In addition, word-level optimization
based on numbers ignores the semantics of reports and medical images, and the
generated reports often cannot achieve good performance. Therefore, we propose
a cross-modal consistent multi-view medical report generation with a domain
transfer network (C^2M-DoT). Specifically, (i) a semantic-based multi-view
contrastive learning medical report generation framework is adopted to utilize
cross-view information to learn the semantic representation of lesions; (ii) a
domain transfer network is further proposed to ensure that the multi-view
report generation model can still achieve good inference performance under
single-view input; (iii) meanwhile, optimization using a cross-modal
consistency loss facilitates the generation of textual reports that are
semantically consistent with medical images. Extensive experimental studies on
two public benchmark datasets demonstrate that C^2M-DoT substantially
outperforms state-of-the-art baselines in all metrics. Ablation studies also
confirmed the validity and necessity of each component in C^2M-DoT.