BCN: Batch Channel Normalization for Image Classification
Afifa Khaled Chao Li Jia Ning Kun He
Normalization techniques have been widely used in the field of deep learning
due to their capability of enabling higher learning rates and are less careful
in initialization. However, the effectiveness of popular normalization
technologies is typically limited to specific areas. Unlike the standard Batch
Normalization (BN) and Layer Normalization (LN), where BN computes the mean and
variance along the (N,H,W) dimensions and LN computes the mean and variance
along the (C,H,W) dimensions (N, C, H and W are the batch, channel, spatial
height and width dimension, respectively), this paper presents a novel
normalization technique called Batch Channel Normalization (BCN). To exploit
both the channel and batch dependence and adaptively and combine the advantages
of BN and LN based on specific datasets or tasks, BCN separately normalizes
inputs along the (N, H, W) and (C, H, W) axes, then combines the normalized
outputs based on adaptive parameters. As a basic block, BCN can be easily
integrated into existing models for various applications in the field of
computer vision. Empirical results show that the proposed technique can be
seamlessly applied to various versions of CNN or Vision Transformer
architecture. The code is publicly available at
https://github.com/AfifaKhaled/BatchChannel-Normalization