Enhancing Efficiency in Vision Transformer Networks: Design Techniques
  and Insights
Moein Heidari Reza Azad Sina Ghorbani Kolahi René Arimond Leon Niggemeier Alaa Sulaiman Afshin Bozorgpour Ehsan Khodapanah Aghdam Amirhossein Kazerouni Ilker Hacihaliloglu Dorit Merhof
Intrigued by the inherent ability of the human visual system to identify
salient regions in complex scenes, attention mechanisms have been seamlessly
integrated into various Computer Vision (CV) tasks. Building upon this
paradigm, Vision Transformer (ViT) networks exploit attention mechanisms for
improved efficiency. This review navigates the landscape of redesigned
attention mechanisms within ViTs, aiming to enhance their performance. This
paper provides a comprehensive exploration of techniques and insights for
designing attention mechanisms, systematically reviewing recent literature in
the field of CV. This survey begins with an introduction to the theoretical
foundations and fundamental concepts underlying attention mechanisms. We then
present a systematic taxonomy of various attention mechanisms within ViTs,
employing redesigned approaches. A multi-perspective categorization is proposed
based on their application, objectives, and the type of attention applied. The
analysis includes an exploration of the novelty, strengths, weaknesses, and an
in-depth evaluation of the different proposed strategies. This culminates in
the development of taxonomies that highlight key properties and contributions.
Finally, we gather the reviewed studies along with their available open-source
implementations at our
\href{https://github.com/mindflow-institue/Awesome-Attention-Mechanism-in-Medical-Imaging}{GitHub}\footnote{\url{https://github.com/xmindflow/Awesome-Attention-Mechanism-in-Medical-Imaging}}.
We aim to regularly update it with the most recent relevant papers.