Pathology-and-genomics Multimodal Transformer for Survival Outcome
  Prediction
Kexin Ding Mu Zhou Dimitris N. Metaxas Shaoting Zhang
Survival outcome assessment is challenging and inherently associated with
multiple clinical factors (e.g., imaging and genomics biomarkers) in cancer.
Enabling multimodal analytics promises to reveal novel predictive patterns of
patient outcomes. In this study, we propose a multimodal transformer
(PathOmics) integrating pathology and genomics insights into colon-related
cancer survival prediction. We emphasize the unsupervised pretraining to
capture the intrinsic interaction between tissue microenvironments in gigapixel
whole slide images (WSIs) and a wide range of genomics data (e.g.,
mRNA-sequence, copy number variant, and methylation). After the multimodal
knowledge aggregation in pretraining, our task-specific model finetuning could
expand the scope of data utility applicable to both multi- and single-modal
data (e.g., image- or genomics-only). We evaluate our approach on both TCGA
colon and rectum cancer cohorts, showing that the proposed approach is
competitive and outperforms state-of-the-art studies. Finally, our approach is
desirable to utilize the limited number of finetuned samples towards
data-efficient analytics for survival outcome prediction. The code is available
at https://github.com/Cassie07/PathOmics.