LoRA Training in the NTK Regime has No Spurious Local Minima
Uijeong Jang Jason D. Lee Ernest K. Ryu
Low-rank adaptation (LoRA) has become the standard approach for
parameter-efficient fine-tuning of large language models (LLM), but our
theoretical understanding of LoRA has been limited. In this work, we
theoretically analyze LoRA fine-tuning in the neural tangent kernel (NTK)
regime with $N$ data points, showing: (i) full fine-tuning (without LoRA)
admits a low-rank solution of rank $r\lesssim \sqrt{N}$; (ii) using LoRA with
rank $r\gtrsim \sqrt{N}$ eliminates spurious local minima, allowing gradient
descent to find the low-rank solutions; (iii) the low-rank solution found using
LoRA generalizes well.