Comprehensive Survey of Model Compression and Speed up for Vision
  Transformers
Feiyang Chen Ziqian Luo Lisang Zhou Xueting Pan Ying Jiang
Vision Transformers (ViT) have marked a paradigm shift in computer vision,
outperforming state-of-the-art models across diverse tasks. However, their
practical deployment is hampered by high computational and memory demands. This
study addresses the challenge by evaluating four primary model compression
techniques: quantization, low-rank approximation, knowledge distillation, and
pruning. We methodically analyze and compare the efficacy of these techniques
and their combinations in optimizing ViTs for resource-constrained
environments. Our comprehensive experimental evaluation demonstrates that these
methods facilitate a balanced compromise between model accuracy and
computational efficiency, paving the way for wider application in edge
computing devices.