A Framework to Implement 1+N Multi-task Fine-tuning Pattern in LLMs
  Using the CGC-LORA Algorithm
Chao Song Zhihao Ye Qiqiang Lin Qiuying Peng Jun Wang
With the productive evolution of large language models (LLMs) in the field of
natural language processing (NLP), tons of effort has been made to effectively
fine-tune common pre-trained LLMs to fulfill a variety of tasks in one or
multiple specific domain. In practice, there are two prevailing ways, in which
the adaptation can be achieved: (i) Multiple Independent Models: Pre-trained
LLMs are fine-tuned a few times independently using the corresponding training
samples from each task. (ii) An Integrated Model: Samples from all tasks are
employed to fine-tune a pre-trianed LLM unitedly. To address the high computing
cost and seesawing issue simultaneously, we propose a unified framework that
implements a 1 + N mutli-task fine-tuning pattern in LLMs using a novel
Customized Gate Control (CGC) Low-rank Adaptation (LoRA) algorithm. Our work
aims to take an advantage of both MTL (i.e., CGC) and PEFT (i.e., LoRA) scheme.
For a given cluster of tasks, we design an innovative layer that contains two
types of experts as additional trainable parameters to make LoRA be compatible
with MTL. To comprehensively evaluate the proposed framework, we conduct
well-designed experiments on two public datasets. The experimental results
demonstrate that the unified framework with CGC-LoRA modules achieves higher
evaluation scores than all benchmarks on both two datasets.