ParFormer: Vision Transformer Baseline with Parallel Local Global Token
  Mixer and Convolution Attention Patch Embedding
Novendra Setyawan Ghufron Wahyu Kurniawan Chi-Chia Sun Jun-Wei Hsieh Hui-Kai Su Wen-Kai Kuo
This work presents ParFormer as an enhanced transformer architecture that
allows the incorporation of different token mixers into a single stage, hence
improving feature extraction capabilities. Integrating both local and global
data allows for precise representation of short- and long-range spatial
relationships without the need for computationally intensive methods such as
shifting windows. Along with the parallel token mixer encoder, We offer the
Convolutional Attention Patch Embedding (CAPE) as an enhancement of standard
patch embedding to improve token mixer extraction with a convolutional
attention module. Our comprehensive evaluation demonstrates that our ParFormer
outperforms CNN-based and state-of-the-art transformer-based architectures in
image classification and several complex tasks such as object recognition. The
proposed CAPE has been demonstrated to benefit the overall MetaFormer
architecture, even while utilizing the Identity Mapping Token Mixer, resulting
in a 0.5\% increase in accuracy. The ParFormer models outperformed ConvNeXt and
Swin Transformer for the pure convolution and transformer model in accuracy.
Furthermore, our model surpasses the current leading hybrid transformer by
reaching competitive Top-1 scores in the ImageNet-1K classification test.
Specifically, our model variants with 11M, 23M, and 34M parameters achieve
scores of 80.4\%, 82.1\%, and 83.1\%, respectively. Code:
https://github.com/novendrastywn/ParFormer-CAPE-2024