EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba
Xiaohuan Pei Tao Huang Chang Xu
Prior efforts in light-weight model development mainly centered on CNN and
Transformer-based designs yet faced persistent challenges. CNNs adept at local
feature extraction compromise resolution while Transformers offer global reach
but escalate computational demands $\mathcal{O}(N^2)$. This ongoing trade-off
between accuracy and efficiency remains a significant hurdle. Recently, state
space models (SSMs), such as Mamba, have shown outstanding performance and
competitiveness in various tasks such as language modeling and computer vision,
while reducing the time complexity of global information extraction to
$\mathcal{O}(N)$. Inspired by this, this work proposes to explore the potential
of visual state space models in light-weight model design and introduce a novel
efficient model variant dubbed EfficientVMamba. Concretely, our EfficientVMamba
integrates a atrous-based selective scan approach by efficient skip sampling,
constituting building blocks designed to harness both global and local
representational features. Additionally, we investigate the integration between
SSM blocks and convolutions, and introduce an efficient visual state space
block combined with an additional convolution branch, which further elevate the
model performance. Experimental results show that, EfficientVMamba scales down
the computational complexity while yields competitive results across a variety
of vision tasks. For example, our EfficientVMamba-S with $1.3$G FLOPs improves
Vim-Ti with $1.5$G FLOPs by a large margin of $5.6\%$ accuracy on ImageNet.
Code is available at: \url{https://github.com/TerryPei/EfficientVMamba}.