TurboViT: Generating Fast Vision Transformers via Generative
  Architecture Search
Alexander Wong Saad Abbasi Saeejith Nair
Vision transformers have shown unprecedented levels of performance in
tackling various visual perception tasks in recent years. However, the
architectural and computational complexity of such network architectures have
made them challenging to deploy in real-world applications with
high-throughput, low-memory requirements. As such, there has been significant
research recently on the design of efficient vision transformer architectures.
In this study, we explore the generation of fast vision transformer
architecture designs via generative architecture search (GAS) to achieve a
strong balance between accuracy and architectural and computational efficiency.
Through this generative architecture search process, we create TurboViT, a
highly efficient hierarchical vision transformer architecture design that is
generated around mask unit attention and Q-pooling design patterns. The
resulting TurboViT architecture design achieves significantly lower
architectural computational complexity (>2.47$\times$ smaller than FasterViT-0
while achieving same accuracy) and computational complexity (>3.4$\times$ fewer
FLOPs and 0.9% higher accuracy than MobileViT2-2.0) when compared to 10 other
state-of-the-art efficient vision transformer network architecture designs
within a similar range of accuracy on the ImageNet-1K dataset. Furthermore,
TurboViT demonstrated strong inference latency and throughput in both
low-latency and batch processing scenarios (>3.21$\times$ lower latency and
>3.18$\times$ higher throughput compared to FasterViT-0 for low-latency
scenario). These promising results demonstrate the efficacy of leveraging
generative architecture search for generating efficient transformer
architecture designs for high-throughput scenarios.