ASPEN: High-Throughput LoRA Fine-Tuning of Large Language Models with a
  Single GPU
Zhengmao Ye Dengchun Li Jingqi Tian Tingfeng Lan Jie Zuo Lei Duan Hui Lu Yexi Jiang Jian Sha Ke Zhang Mingjie Tang
Transformer-based large language models (LLMs) have demonstrated outstanding
performance across diverse domains, particularly when fine-turned for specific
domains. Recent studies suggest that the resources required for fine-tuning
LLMs can be economized through parameter-efficient methods such as Low-Rank
Adaptation (LoRA). While LoRA effectively reduces computational burdens and
resource demands, it currently supports only a single-job fine-tuning setup.
  In this paper, we present ASPEN, a high-throughput framework for fine-tuning
LLMs. ASPEN efficiently trains multiple jobs on a single GPU using the LoRA
method, leveraging shared pre-trained model and adaptive scheduling. ASPEN is
compatible with transformer-based language models like LLaMA and ChatGLM, etc.
Experiments show that ASPEN saves 53% of GPU memory when training multiple
LLaMA-7B models on NVIDIA A100 80GB GPU and boosts training throughput by about
17% compared to existing methods when training with various pre-trained models
on different GPUs. The adaptive scheduling algorithm reduces turnaround time by
24%, end-to-end training latency by 12%, prioritizing jobs and preventing
out-of-memory issues.