Improving Discriminative Multi-Modal Learning with Large-Scale
  Pre-Trained Models
Chenzhuang Du Yue Zhao Chonghua Liao Jiacheng You Jie Fu Hang Zhao
This paper investigates how to better leverage large-scale pre-trained
uni-modal models to further enhance discriminative multi-modal learning. Even
when fine-tuned with only uni-modal data, these models can outperform previous
multi-modal models in certain tasks. It's clear that their incorporation into
multi-modal learning would significantly improve performance. However,
multi-modal learning with these models still suffers from insufficient learning
of uni-modal features, which weakens the resulting multi-modal model's
generalization ability. While fine-tuning uni-modal models separately and then
aggregating their predictions is straightforward, it doesn't allow for adequate
adaptation between modalities, also leading to sub-optimal results. To this
end, we introduce Multi-Modal Low-Rank Adaptation learning (MMLoRA). By
freezing the weights of uni-modal fine-tuned models, adding extra trainable
rank decomposition matrices to them, and subsequently performing multi-modal
joint training, our method enhances adaptation between modalities and boosts
overall performance. We demonstrate the effectiveness of MMLoRA on three
dataset categories: audio-visual (e.g., AVE, Kinetics-Sound, CREMA-D),
vision-language (e.g., MM-IMDB, UPMC Food101), and RGB-Optical Flow (UCF101).