Spectral Convolutional Transformer: Harmonizing Real vs. Complex
  Multi-View Spectral Operators for Vision Transformer
Badri N. Patro Vinay P. Namboodiri Vijay S. Agneeswaran
Transformers used in vision have been investigated through diverse
architectures - ViT, PVT, and Swin. These have worked to improve the attention
mechanism and make it more efficient. Differently, the need for including local
information was felt, leading to incorporating convolutions in transformers
such as CPVT and CvT. Global information is captured using a complex Fourier
basis to achieve global token mixing through various methods, such as AFNO,
GFNet, and Spectformer. We advocate combining three diverse views of data -
local, global, and long-range dependence. We also investigate the simplest
global representation using only the real domain spectral representation -
obtained through the Hartley transform. We use a convolutional operator in the
initial layers to capture local information. Through these two contributions,
we are able to optimize and obtain a spectral convolution transformer (SCT)
that provides improved performance over the state-of-the-art methods while
reducing the number of parameters. Through extensive experiments, we show that
SCT-C-small gives state-of-the-art performance on the ImageNet dataset and
reaches 84.5\% top-1 accuracy, while SCT-C-Large reaches 85.9\% and SCT-C-Huge
reaches 86.4\%. We evaluate SCT on transfer learning on datasets such as
CIFAR-10, CIFAR-100, Oxford Flower, and Stanford Car. We also evaluate SCT on
downstream tasks i.e. instance segmentation on the MSCOCO dataset. The project
page is available on this webpage.\url{https://github.com/badripatro/sct}