ASVD: Activation-aware Singular Value Decomposition for Compressing
  Large Language Models
Zhihang Yuan Yuzhang Shang Yue Song Qiang Wu Yan Yan Guangyu Sun
This paper explores a new post-hoc training-free compression paradigm for
compressing Large Language Models (LLMs) to facilitate their wider adoption in
various computing environments. We delve into the challenges of LLM
compression, notably their dependency on extensive training data and
computational resources. We propose a training-free approach dubbed
Activation-aware Singular Value Decomposition (ASVD) to address these
limitations. ASVD effectively manages activation outliers by adjusting the
weight matrix based on the activation distribution, improving decomposition
accuracy and efficiency. Our method also addresses the varying sensitivity of
different LLM layers to decomposition, with an iterative calibration process
for optimal layer-specific decomposition. Experiments demonstrate that ASVD can
compress network by 10%-20% without losing reasoning capacities. Additionally,
it can be seamlessly integrated with other LLM compression paradigms,
showcasing its flexible compatibility. Code and compressed models are available
at https://github.com/hahnyuan/ASVD4LLM.