Flora: Low-Rank Adapters Are Secretly Gradient Compressors
Yongchang Hao Yanshuai Cao Lili Mou
Despite large neural networks demonstrating remarkable abilities to complete
different tasks, they require excessive memory usage to store the optimization
states for training. To alleviate this, the low-rank adaptation (LoRA) is
proposed to reduce the optimization states by training fewer parameters.
However, LoRA restricts overall weight update matrices to be low-rank, limiting
the model performance. In this work, we investigate the dynamics of LoRA and
identify that it can be approximated by a random projection. Based on this
observation, we propose Flora, which is able to achieve high-rank updates by
resampling the projection matrices while enjoying the sublinear space
complexity of optimization states. We conduct experiments across different
tasks and model architectures to verify the effectiveness of our approach.