Run LoRA Run: Faster and Lighter LoRA Implementations
Daria Cherniuk Aleksandr Mikhalev Ivan Oseledets
LoRA is a technique that reduces the number of trainable parameters in a
neural network by introducing low-rank adapters to linear layers. This
technique is used both for fine-tuning (LoRA, QLoRA) and full train (ReLoRA).
This paper presents the RunLoRA framework for efficient implementations of LoRA
that significantly improves the speed of neural network training and
fine-tuning using low-rank adapters. The proposed implementation optimizes the
computation of LoRA operations based on dimensions of corresponding linear
layer, layer input dimensions and lora rank by choosing best forward and
backward computation graph based on FLOPs and time estimations, resulting in
faster training without sacrificing accuracy. The experimental results show up
to 17% speedup on Llama family of models.