LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative
  Tasks
Hanqing Wang Bowen Ping Shuo Wang Xu Han Yun Chen Zhiyuan Liu Maosong Sun
LoRA employs lightweight modules to customize large language models (LLMs)
for each downstream task or domain, where different learned additional modules
represent diverse skills. Combining existing LoRAs to address new tasks can
enhance the reusability of learned LoRAs, particularly beneficial for tasks
with limited annotated data. Most prior works on LoRA combination primarily
rely on task-level weights for each involved LoRA, making different examples
and tokens share the same LoRA weights. However, in generative tasks, different
tokens may necessitate diverse skills to manage. Taking the Chinese math task
as an example, understanding the problem description may depend more on the
Chinese LoRA, while the calculation part may rely more on the math LoRA. To
this end, we propose LoRA-Flow, which utilizes dynamic weights to adjust the
impact of different LoRAs. The weights at each step are determined by a fusion
gate with extremely few parameters, which can be learned with only 200 training
examples. Experiments across six generative tasks demonstrate that our method
consistently outperforms baselines with task-level fusion weights. This
underscores the necessity of introducing dynamic fusion weights for LoRA
combination.