Shift-ConvNets: Small Convolutional Kernel with Large Kernel Effects
Dachong Li Li Li Zhuangzhuang Chen Jianqiang Li
Recent studies reveal that the remarkable performance of Vision transformers
(ViTs) benefits from large receptive fields. For this reason, the large
convolutional kernel design becomes an ideal solution to make Convolutional
Neural Networks (CNNs) great again. However, the typical large convolutional
kernels turn out to be hardware-unfriendly operators, resulting in discount
compatibility of various hardware platforms. Thus, it is unwise to simply
enlarge the convolutional kernel size. In this paper, we reveal that small
convolutional kernels and convolution operations can achieve the closing
effects of large kernel sizes. Then, we propose a shift-wise operator that
ensures the CNNs capture long-range dependencies with the help of the sparse
mechanism, while remaining hardware-friendly. Experimental results show that
our shift-wise operator significantly improves the accuracy of a regular CNN
while markedly reducing computational requirements. On the ImageNet-1k, our
shift-wise enhanced CNN model outperforms the state-of-the-art models. Code &
models at https://github.com/lidc54/shift-wiseConv.