SPFormer: Enhancing Vision Transformer with Superpixel Representation
Jieru Mei Liang-Chieh Chen Alan Yuille Cihang Xie
In this work, we introduce SPFormer, a novel Vision Transformer enhanced by
superpixel representation. Addressing the limitations of traditional Vision
Transformers' fixed-size, non-adaptive patch partitioning, SPFormer employs
superpixels that adapt to the image's content. This approach divides the image
into irregular, semantically coherent regions, effectively capturing intricate
details and applicable at both initial and intermediate feature levels.
  SPFormer, trainable end-to-end, exhibits superior performance across various
benchmarks. Notably, it exhibits significant improvements on the challenging
ImageNet benchmark, achieving a 1.4% increase over DeiT-T and 1.1% over DeiT-S
respectively. A standout feature of SPFormer is its inherent explainability.
The superpixel structure offers a window into the model's internal processes,
providing valuable insights that enhance the model's interpretability. This
level of clarity significantly improves SPFormer's robustness, particularly in
challenging scenarios such as image rotations and occlusions, demonstrating its
adaptability and resilience.