ELA: Efficient Local Attention for Deep Convolutional Neural Networks
Wei Xu Yi Wan
The attention mechanism has gained significant recognition in the field of
computer vision due to its ability to effectively enhance the performance of
deep neural networks. However, existing methods often struggle to effectively
utilize spatial information or, if they do, they come at the cost of reducing
channel dimensions or increasing the complexity of neural networks. In order to
address these limitations, this paper introduces an Efficient Local Attention
(ELA) method that achieves substantial performance improvements with a simple
structure. By analyzing the limitations of the Coordinate Attention method, we
identify the lack of generalization ability in Batch Normalization, the adverse
effects of dimension reduction on channel attention, and the complexity of
attention generation process. To overcome these challenges, we propose the
incorporation of 1D convolution and Group Normalization feature enhancement
techniques. This approach enables accurate localization of regions of interest
by efficiently encoding two 1D positional feature maps without the need for
dimension reduction, while allowing for a lightweight implementation. We
carefully design three hyperparameters in ELA, resulting in four different
versions: ELA-T, ELA-B, ELA-S, and ELA-L, to cater to the specific requirements
of different visual tasks such as image classification, object detection and
sementic segmentation. ELA can be seamlessly integrated into deep CNN networks
such as ResNet, MobileNet, and DeepLab. Extensive evaluations on the ImageNet,
MSCOCO, and Pascal VOC datasets demonstrate the superiority of the proposed ELA
module over current state-of-the-art methods in all three aforementioned visual
tasks.