MoPE: Parameter-Efficient and Scalable Multimodal Fusion via Mixture of
  Prompt Experts
Ruixiang Jiang Lingbo Liu Changwen Chen
Prompt-tuning has demonstrated parameter-efficiency in fusing unimodal
foundation models for multimodal tasks. However, its limited adaptivity and
expressiveness lead to suboptimal performance when compared with other tuning
methods. In this paper, we address this issue by disentangling the vanilla
prompts to adaptively capture dataset-level and instance-level features.
Building upon this disentanglement, we introduce the mixture of prompt experts
(MoPE) technique to enhance expressiveness. MoPE leverages multimodal pairing
priors to route the most effective prompt on a per-instance basis. Compared to
vanilla prompting, our MoPE-based conditional prompting exhibits greater
expressiveness for multimodal fusion, scaling better with the training data and
the overall number of trainable parameters. We also study a regularization term
for expert routing, leading to emergent expert specialization, where different
experts focus on different concepts, enabling interpretable soft prompting.
Extensive experiments across three multimodal datasets demonstrate that our
method achieves state-of-the-art results, matching or even surpassing the
performance of fine-tuning, while requiring only 0.8% of the trainable
parameters. Code will be released: https://github.com/songrise/MoPE.