Sparse then Prune: Toward Efficient Vision Transformers
Yogi Prasetyo Novanto Yudistira Agus Wahyu Widodo
The Vision Transformer architecture is a deep learning model inspired by the
success of the Transformer model in Natural Language Processing. However, the
self-attention mechanism, large number of parameters, and the requirement for a
substantial amount of training data still make Vision Transformers
computationally burdensome. In this research, we investigate the possibility of
applying Sparse Regularization to Vision Transformers and the impact of
Pruning, either after Sparse Regularization or without it, on the trade-off
between performance and efficiency. To accomplish this, we apply Sparse
Regularization and Pruning methods to the Vision Transformer architecture for
image classification tasks on the CIFAR-10, CIFAR-100, and ImageNet-100
datasets. The training process for the Vision Transformer model consists of two
parts: pre-training and fine-tuning. Pre-training utilizes ImageNet21K data,
followed by fine-tuning for 20 epochs. The results show that when testing with
CIFAR-100 and ImageNet-100 data, models with Sparse Regularization can increase
accuracy by 0.12%. Furthermore, applying pruning to models with Sparse
Regularization yields even better results. Specifically, it increases the
average accuracy by 0.568% on CIFAR-10 data, 1.764% on CIFAR-100, and 0.256% on
ImageNet-100 data compared to pruning models without Sparse Regularization.
Code can be accesed here: https://github.com/yogiprsty/Sparse-ViT