An FPGA-Based Accelerator Enabling Efficient Support for CNNs with
  Arbitrary Kernel Sizes
Miaoxin Wang Xiao Wu Jun Lin Zhongfeng Wang
Convolutional neural networks (CNNs) with large kernels, drawing inspiration
from the key operations of vision transformers (ViTs), have demonstrated
impressive performance in various vision-based applications. To address the
issue of computational efficiency degradation in existing designs for
supporting large-kernel convolutions, an FPGA-based inference accelerator is
proposed for the efficient deployment of CNNs with arbitrary kernel sizes.
Firstly, a Z-flow method is presented to optimize the computing data flow by
maximizing data reuse opportunity. Besides, the proposed design, incorporating
the kernel-segmentation (Kseg) scheme, enables extended support for
large-kernel convolutions, significantly reducing the storage requirements for
overlapped data. Moreover, based on the analysis of typical block structures in
emerging CNNs, vertical-fused (VF) and horizontal-fused (HF) methods are
developed to optimize CNN deployments from both computation and transmission
perspectives. The proposed hardware accelerator, evaluated on Intel Arria 10
FPGA, achieves up to 3.91 times better DSP efficiency than prior art on the
same network. Particularly, it demonstrates efficient support for large-kernel
CNNs, achieving throughputs of 169.68 GOPS and 244.55 GOPS for RepLKNet-31 and
PyConvResNet-50, respectively, both of which are implemented on hardware for
the first time.