An FPGA-Based Reconfigurable Accelerator for Convolution-Transformer
  Hybrid EfficientViT
Haikuo Shao Huihong Shi Wendong Mao Zhongfeng Wang
Vision Transformers (ViTs) have achieved significant success in computer
vision. However, their intensive computations and massive memory footprint
challenge ViTs' deployment on embedded devices, calling for efficient ViTs.
Among them, EfficientViT, the state-of-the-art one, features a
Convolution-Transformer hybrid architecture, enhancing both accuracy and
hardware efficiency. Unfortunately, existing accelerators cannot fully exploit
the hardware benefits of EfficientViT due to its unique architecture. In this
paper, we propose an FPGA-based accelerator for EfficientViT to advance the
hardware efficiency frontier of ViTs. Specifically, we design a reconfigurable
architecture to efficiently support various operation types, including
lightweight convolutions and attention, boosting hardware utilization.
Additionally, we present a time-multiplexed and pipelined dataflow to
facilitate both intra- and inter-layer fusions, reducing off-chip data access
costs. Experimental results show that our accelerator achieves up to 780.2 GOPS
in throughput and 105.1 GOPS/W in energy efficiency at 200MHz on the Xilinx
ZCU102 FPGA, which significantly outperforms prior works.