Transformer Compression via Subspace Projection
Yuxuan Hu Jing Zhang Chen Zhao Cuiping Li Hong Chen
We propose TCSP, a novel method for compressing a transformer model by
focusing on reducing the hidden size of the model. By projecting the whole
transform model into a subspace, we enable matrix operations between the weight
matrices in the model and features in a reduced-dimensional space, leading to
significant reductions in model parameters and computing resources. To
establish this subspace, we decompose the feature matrix, derived from
different layers of sampled data instances, into a projection matrix. For
evaluation, TCSP is applied to compress T5 and BERT models on the GLUE and
SQuAD benchmarks. Experimental results demonstrate that TCSP achieves a
compression ratio of 44\% with at most 1.6\% degradation in accuracy,
surpassing or matching prior compression methods. Furthermore, TCSP exhibits
compatibility with other methods targeting filter and attention head size
compression.