Vision Transformer Pruning Via Matrix Decomposition
Tianyi Sun
This is a further development of Vision Transformer Pruning via matrix
decomposition. The purpose of the Vision Transformer Pruning is to prune the
dimension of the linear projection of the dataset by learning their associated
importance score in order to reduce the storage, run-time memory, and
computational demands. In this paper we further reduce dimension and complexity
of the linear projection by implementing and comparing several matrix
decomposition methods while preserving the generated important features. We end
up selected the Singular Value Decomposition as the method to achieve our goal
by comparing the original accuracy scores in the original Github repository and
the accuracy scores of using those matrix decomposition methods, including
Singular Value Decomposition, four versions of QR Decomposition, and LU
factorization.