FMViT: A multiple-frequency mixing Vision Transformer
Wei Tan Yifeng Geng Xuansong Xie
The transformer model has gained widespread adoption in computer vision tasks
in recent times. However, due to the quadratic time and memory complexity of
self-attention, which is proportional to the number of input tokens, most
existing Vision Transformers (ViTs) encounter challenges in achieving efficient
performance in practical industrial deployment scenarios, such as TensorRT and
CoreML, where traditional CNNs excel. Although some recent attempts have been
made to design CNN-Transformer hybrid architectures to tackle this problem,
their overall performance has not met expectations. To tackle these challenges,
we propose an efficient hybrid ViT architecture named FMViT. This approach
enhances the model's expressive power by blending high-frequency features and
low-frequency features with varying frequencies, enabling it to capture both
local and global information effectively. Additionally, we introduce
deploy-friendly mechanisms such as Convolutional Multigroup Reparameterization
(gMLP), Lightweight Multi-head Self-Attention (RLMHSA), and Convolutional
Fusion Block (CFB) to further improve the model's performance and reduce
computational overhead. Our experiments demonstrate that FMViT surpasses
existing CNNs, ViTs, and CNNTransformer hybrid architectures in terms of
latency/accuracy trade-offs for various vision tasks. On the TensorRT platform,
FMViT outperforms Resnet101 by 2.5% (83.3% vs. 80.8%) in top-1 accuracy on the
ImageNet dataset while maintaining similar inference latency. Moreover, FMViT
achieves comparable performance with EfficientNet-B5, but with a 43%
improvement in inference speed. On CoreML, FMViT outperforms MobileOne by 2.6%
in top-1 accuracy on the ImageNet dataset, with inference latency comparable to
MobileOne (78.5% vs. 75.9%). Our code can be found at
https://github.com/tany0699/FMViT.