CageViT: Convolutional Activation Guided Efficient Vision Transformer
Hao Zheng Jinbao Wang Xiantong Zhen Hong Chen Jingkuan Song Feng Zheng
Recently, Transformers have emerged as the go-to architecture for both vision
and language modeling tasks, but their computational efficiency is limited by
the length of the input sequence. To address this, several efficient variants
of Transformers have been proposed to accelerate computation or reduce memory
consumption while preserving performance. This paper presents an efficient
vision Transformer, called CageViT, that is guided by convolutional activation
to reduce computation. Our CageViT, unlike current Transformers, utilizes a new
encoder to handle the rearranged tokens, bringing several technical
contributions: 1) Convolutional activation is used to pre-process the token
after patchifying the image to select and rearrange the major tokens and minor
tokens, which substantially reduces the computation cost through an additional
fusion layer. 2) Instead of using the class activation map of the convolutional
model directly, we design a new weighted class activation to lower the model
requirements. 3) To facilitate communication between major tokens and fusion
tokens, Gated Linear SRA is proposed to further integrate fusion tokens into
the attention mechanism. We perform a comprehensive validation of CageViT on
the image classification challenge.
  Experimental results demonstrate that the proposed CageViT outperforms the
most recent state-of-the-art backbones by a large margin in terms of
efficiency, while maintaining a comparable level of accuracy (e.g. a
moderate-sized 43.35M model trained solely on 224 x 224 ImageNet-1K can achieve
Top-1 accuracy of 83.4% accuracy).