Ternary Singular Value Decomposition as a Better Parameterized Form in
  Linear Mapping
Boyu Chen Hanxuan Chen Jiao He Fengyu Sun Shangling Jui
We present a simple yet novel parameterized form of linear mapping to
achieves remarkable network compression performance: a pseudo SVD called
Ternary SVD (TSVD).
  Unlike vanilla SVD, TSVD limits the $U$ and $V$ matrices in SVD to ternary
matrices form in $\{\pm 1, 0\}$. This means that instead of using the expensive
multiplication instructions, TSVD only requires addition instructions when
computing $U(\cdot)$ and $V(\cdot)$.
  We provide direct and training transition algorithms for TSVD like Post
Training Quantization and Quantization Aware Training respectively.
Additionally, we analyze the convergence of the direct transition algorithms in
theory.
  In experiments, we demonstrate that TSVD can achieve state-of-the-art network
compression performance in various types of networks and tasks, including
current baseline models such as ConvNext, Swim, BERT, and large language model
like OPT.