MOELoRA: An MOE-based Parameter Efficient Fine-Tuning Method for
  Multi-task Medical Applications
Qidong Liu Xian Wu Xiangyu Zhao Yuanshao Zhu Derong Xu Feng Tian Yefeng Zheng
The recent surge in the field of Large Language Models (LLMs) has gained
significant attention in numerous domains. In order to tailor an LLM to a
specific domain such as a web-based healthcare system, fine-tuning with domain
knowledge is necessary. However, two issues arise during fine-tuning LLMs for
medical applications. The first is the problem of task variety, where there are
numerous distinct tasks in real-world medical scenarios. This diversity often
results in suboptimal fine-tuning due to data imbalance and seesawing problems.
Additionally, the high cost of fine-tuning can be prohibitive, impeding the
application of LLMs. The large number of parameters in LLMs results in enormous
time and computational consumption during fine-tuning, which is difficult to
justify. To address these two issues simultaneously, we propose a novel
parameter-efficient fine-tuning framework for multi-task medical applications
called MOELoRA. The framework aims to capitalize on the benefits of both MOE
for multi-task learning and LoRA for parameter-efficient fine-tuning. To unify
MOE and LoRA, we devise multiple experts as the trainable parameters, where
each expert consists of a pair of low-rank matrices to maintain a small number
of trainable parameters. Additionally, we propose a task-motivated gate
function for all MOELoRA layers that can regulate the contributions of each
expert and generate distinct parameters for various tasks. To validate the
effectiveness and practicality of the proposed method, we conducted
comprehensive experiments on a public multi-task Chinese medical dataset. The
experimental results demonstrate that MOELoRA outperforms existing
parameter-efficient fine-tuning methods. The implementation is available online
for convenient reproduction of our experiments.