TransAxx: Efficient Transformers with Approximate Computing
Dimitrios Danopoulos Georgios Zervakis Dimitrios Soudris Jörg Henkel
Vision Transformer (ViT) models which were recently introduced by the
transformer architecture have shown to be very competitive and often become a
popular alternative to Convolutional Neural Networks (CNNs). However, the high
computational requirements of these models limit their practical applicability
especially on low-power devices. Current state-of-the-art employs approximate
multipliers to address the highly increased compute demands of DNN accelerators
but no prior research has explored their use on ViT models. In this work we
propose TransAxx, a framework based on the popular PyTorch library that enables
fast inherent support for approximate arithmetic to seamlessly evaluate the
impact of approximate computing on DNNs such as ViT models. Using TransAxx we
analyze the sensitivity of transformer models on the ImageNet dataset to
approximate multiplications and perform approximate-aware finetuning to regain
accuracy. Furthermore, we propose a methodology to generate approximate
accelerators for ViT models. Our approach uses a Monte Carlo Tree Search (MCTS)
algorithm to efficiently search the space of possible configurations using a
hardware-driven hand-crafted policy. Our evaluation demonstrates the efficacy
of our methodology in achieving significant trade-offs between accuracy and
power, resulting in substantial gains without compromising on performance.