TTMFN: Two-stream Transformer-based Multimodal Fusion Network for
  Survival Prediction
Ruiquan Ge Xiangyang Hu Rungen Huang Gangyong Jia Yaqi Wang Renshu Gu Changmiao Wang Elazab Ahmed Linyan Wang Juan Ye Ye Li
Survival prediction plays a crucial role in assisting clinicians with the
development of cancer treatment protocols. Recent evidence shows that
multimodal data can help in the diagnosis of cancer disease and improve
survival prediction. Currently, deep learning-based approaches have experienced
increasing success in survival prediction by integrating pathological images
and gene expression data. However, most existing approaches overlook the
intra-modality latent information and the complex inter-modality correlations.
Furthermore, existing modalities do not fully exploit the immense
representational capabilities of neural networks for feature aggregation and
disregard the importance of relationships between features. Therefore, it is
highly recommended to address these issues in order to enhance the prediction
performance by proposing a novel deep learning-based method. We propose a novel
framework named Two-stream Transformer-based Multimodal Fusion Network for
survival prediction (TTMFN), which integrates pathological images and gene
expression data. In TTMFN, we present a two-stream multimodal co-attention
transformer module to take full advantage of the complex relationships between
different modalities and the potential connections within the modalities.
Additionally, we develop a multi-head attention pooling approach to effectively
aggregate the feature representations of the two modalities. The experiment
results on four datasets from The Cancer Genome Atlas demonstrate that TTMFN
can achieve the best performance or competitive results compared to the
state-of-the-art methods in predicting the overall survival of patients.