LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot
  Compression
Ayush Kaushal Tejas Vaidhya Irina Rish
Low Rank Decomposition of matrix - splitting a large matrix into a product of
two smaller matrix offers a means for compression that reduces the parameters
of a model without sparsification, and hence delivering more speedup on modern
hardware. Moreover, unlike quantization, the compressed linear layers remain
fully differentiable and all the parameters trainable, while being able to
leverage the existing highly efficient kernels over floating point matrices. We
study the potential to compress Large Language Models (LLMs) for monolingual
Code generation via Low Rank Decomposition (LoRD) and observe that ranks for
the linear layers in these models can be reduced by upto 39.58% with less than
1% increase in perplexity. We then use Low Rank Decomposition (LoRD) to
compress StarCoder 16B to 13.2B parameter with no drop and to 12.3B with
minimal drop in HumanEval Pass@1 score, in less than 10 minutes on a single
A100. The compressed models speeds up inference by up to 22.35% with just a
single line of change in code over huggingface's implementation with pytorch
backend. Low Rank Decomposition (LoRD) models remain compatible with state of
the art near-lossless quantization method such as SpQR, which allows leveraging
further compression gains of quantization. Lastly, QLoRA over Low Rank
Decomposition (LoRD) model further reduces memory requirements by as much as
21.2% over vanilla QLoRA while offering similar gains from parameter efficient
fine tuning. Our work shows Low Rank Decomposition (LoRD) as a promising new
paradigm for LLM compression.