NOLA: Networks as Linear Combination of Low Rank Random Basis
Soroush Abbasi Koohpayegani KL Navaneet Parsa Nooralinejad Soheil Kolouri Hamed Pirsiavash
Large Language Models (LLMs) have recently gained popularity due to their
impressive few-shot performance across various downstream tasks. However,
fine-tuning all parameters and storing a unique model for each downstream task
or domain becomes impractical because of the massive size of checkpoints (e.g.,
350GB in GPT-3). Current literature, such as LoRA, showcases the potential of
low-rank modifications to the original weights of an LLM, enabling efficient
adaptation and storage for task-specific models. These methods can reduce the
number of parameters needed to fine-tune an LLM by several orders of magnitude.
Yet, these methods face two primary limitations: 1) the parameter reduction is
lower-bounded by the rank one decomposition, and 2) the extent of reduction is
heavily influenced by both the model architecture and the chosen rank. For
instance, in larger models, even a rank one decomposition might exceed the
number of parameters truly needed for adaptation. In this paper, we introduce
NOLA, which overcomes the rank one lower bound present in LoRA. It achieves
this by re-parameterizing the low-rank matrices in LoRA using linear
combinations of randomly generated matrices (basis) and optimizing the linear
mixture coefficients only. This approach allows us to decouple the number of
trainable parameters from both the choice of rank and the network architecture.
We present adaptation results using GPT-2 and ViT in natural language and
computer vision tasks. NOLA performs as well as, or better than models with
equivalent parameter counts. Furthermore, we demonstrate that we can halve the
parameters in larger models compared to LoRA with rank one, without sacrificing
performance.