Efficient Low-rank Backpropagation for Vision Transformer Adaptation
Yuedong Yang Hung-Yueh Chiang Guihong Li Diana Marculescu Radu Marculescu
The increasing scale of vision transformers (ViT) has made the efficient
fine-tuning of these large models for specific needs a significant challenge in
various applications. This issue originates from the computationally demanding
matrix multiplications required during the backpropagation process through
linear layers in ViT. In this paper, we tackle this problem by proposing a new
Low-rank BackPropagation via Walsh-Hadamard Transformation (LBP-WHT) method.
Intuitively, LBP-WHT projects the gradient into a low-rank space and carries
out backpropagation. This approach substantially reduces the computation needed
for adapting ViT, as matrix multiplication in the low-rank space is far less
resource-intensive. We conduct extensive experiments with different models
(ViT, hybrid convolution-ViT model) on multiple datasets to demonstrate the
effectiveness of our method. For instance, when adapting an EfficientFormer-L1
model on CIFAR100, our LBP-WHT achieves 10.4% higher accuracy than the
state-of-the-art baseline, while requiring 9 MFLOPs less computation. As the
first work to accelerate ViT adaptation with low-rank backpropagation, our
LBP-WHT method is complementary to many prior efforts and can be combined with
them for better performance.