Unified Multi-modal Diagnostic Framework with Reconstruction
  Pre-training and Heterogeneity-combat Tuning
Yupei Zhang Li Pan Qiushi Yang Tan Li Zhen Chen
Medical multi-modal pre-training has revealed promise in computer-aided
diagnosis by leveraging large-scale unlabeled datasets. However, existing
methods based on masked autoencoders mainly rely on data-level reconstruction
tasks, but lack high-level semantic information. Furthermore, two significant
heterogeneity challenges hinder the transfer of pre-trained knowledge to
downstream tasks, \textit{i.e.}, the distribution heterogeneity between
pre-training data and downstream data, and the modality heterogeneity within
downstream data. To address these challenges, we propose a Unified Medical
Multi-modal Diagnostic (UMD) framework with tailored pre-training and
downstream tuning strategies. Specifically, to enhance the representation
abilities of vision and language encoders, we propose the Multi-level
Reconstruction Pre-training (MR-Pretrain) strategy, including a feature-level
and data-level reconstruction, which guides models to capture the semantic
information from masked inputs of different modalities. Moreover, to tackle two
kinds of heterogeneities during the downstream tuning, we present the
heterogeneity-combat downstream tuning strategy, which consists of a
Task-oriented Distribution Calibration (TD-Calib) and a Gradient-guided
Modality Coordination (GM-Coord). In particular, TD-Calib fine-tunes the
pre-trained model regarding the distribution of downstream datasets, and
GM-Coord adjusts the gradient weights according to the dynamic optimization
status of different modalities. Extensive experiments on five public medical
datasets demonstrate the effectiveness of our UMD framework, which remarkably
outperforms existing approaches on three kinds of downstream tasks.