Efficient Modulation for Vision Networks
Xu Ma Xiyang Dai Jianwei Yang Bin Xiao Yinpeng Chen Yun Fu Lu Yuan
In this work, we present efficient modulation, a novel design for efficient
vision networks. We revisit the modulation mechanism, which operates input
through convolutional context modeling and feature projection layers, and fuses
features via element-wise multiplication and an MLP block. We demonstrate that
the modulation mechanism is particularly well suited for efficient networks and
further tailor the modulation design by proposing the efficient modulation
(EfficientMod) block, which is considered the essential building block for our
networks. Benefiting from the prominent representational ability of modulation
mechanism and the proposed efficient design, our network can accomplish better
trade-offs between accuracy and efficiency and set new state-of-the-art
performance in the zoo of efficient networks. When integrating EfficientMod
with the vanilla self-attention block, we obtain the hybrid architecture which
further improves the performance without loss of efficiency. We carry out
comprehensive experiments to verify EfficientMod's performance. With fewer
parameters, our EfficientMod-s performs 0.6 top-1 accuracy better than
EfficientFormerV2-s2 and is 25% faster on GPU, and 2.9 better than
MobileViTv2-1.0 at the same GPU latency. Additionally, our method presents a
notable improvement in downstream tasks, outperforming EfficientFormerV2-s by
3.6 mIoU on the ADE20K benchmark. Code and checkpoints are available at
https://github.com/ma-xu/EfficientMod.