DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large
  Language Model
Chao Gao Sai Qian Zhang
To enhance the performance of large language models (LLM) on downstream
tasks, one solution is to fine-tune certain LLM parameters and make it better
align with the characteristics of the training dataset. This process is
commonly known as parameter-efficient fine-tuning (PEFT). Due to the scale of
LLM, PEFT operations are usually executed in the public environment (e.g.,
cloud server). This necessitates the sharing of sensitive user data across
public environments, thereby raising potential privacy concerns. To tackle
these challenges, we propose a distributed PEFT framework called DLoRA. DLoRA
enables scalable PEFT operations to be performed collaboratively between the
cloud and user devices. Coupled with the proposed Kill and Revive algorithm,
the evaluation results demonstrate that DLoRA can significantly reduce the
computation and communication workload over the user devices while achieving
superior accuracy and privacy protection.