We conduct a mathematical analysis on the Batch normalization (BN) effect on gradient backpropagation in residual network training in this work, which is believed to play a critical role in addressing the gradient vanishing/explosion problem. Specifically, by analyzing the mean and variance behavior of the input and the gradient in the forward and backward passes through the BN and residual branches, respectively, we show that they work together to confine the gradient variance to a certain range across residual blocks in backpropagation. As a result, the gradient vanishing/explosion problem is avoided. Furthermore, we use the same analysis to discuss the tradeoff between depth and width of a residual network and demonstrate that shallower yet wider resnets have stronger learning performance than deeper yet thinner resnets. Convolutional neural networks (CNNs) BID10 BID1 BID8 aim at learning a feature hierarchy where higher level features are formed by the composition of lower level features. The deep neural networks act as stacked networks with each layer depending on its previous layer's output. The stochastic gradient descent (SGD) method BID12 has proved to be an effective way in training deep networks. The training proceeds in steps with SGD, where a mini-batch from a given dataset is fed at each training step. However, one factor that slows down the stochastic-gradient-based learning of neural networks is the internal covariate shift. It is defined as the change in the distribution of network activations due to the change in network parameters during the training.To improve training efficiency, BID7 introduced a batch normalization (BN) procedure to reduce the internal covariate shift. The BN changes the distribution of each input element at each layer. Let x = (x 1 , x 2 , · · · , x K ), be a K-dimensional input to a layer. The BN first normalizes each dimension of x as DISPLAYFORM0 and then provide the following new input to the layer DISPLAYFORM1 where k = 1, · · · , K and γ k and β k are parameters to be determined. BID7 offered a complete analysis on the BN effect along the forward pass. However, there was little discussion on the BN effect on the backpropagated gradient along the backward pass. This was stated as an open research problem in BID7 . Here, to address this problem, we conduct a mathematical analysis on gradient propagation in batch normalized networks.The number of layers is an important parameter in the neural network design. The training of deep networks has been largely addressed by normalized initialization BID12 BID3 BID11 and intermediate normalization layers BID7 . These techniques enable networks consisting of tens of layers to converge using the SGD in backpropagation. On the other hand, it is observed that the accuracy of conventional CNNs gets saturated and then degrades rapidly as the network layer increases. Such degradation is not caused by over-fitting since adding more layers to a suitably deep model often results in higher training errors BID13 . To address this issue, BID6 introduced the concept of residual branches. A residual network is a stack of residual blocks, where each residual block fits a residual mapping rather than the direct input-output mapping. A similar network, called the highway network, was introduced by BID13 . Being inspired by the LSTM model BID2 , the highway network has additional gates in the shortcut branches of each block.There are two major contributions in this work. First, we propose a mathematical model to analyze the BN effect on gradient propogation in the training of residual networks. It is shown that residual networks perform better than conventional neural networks because residual branches and BN help maintain the gradient variation within a range throughout the training process, thus stabilizing gradient-based-learning of the network. They act as a check on the gradients passing through the network during backpropagation so as to avoid gradient vanishing or explosion. Second, we provide insights into wide residual networks based on the same mathematical analysis. The wide residual network was recently introduced by BID16 . As the gradient goes through the residual network, the network may not learn anything useful since there is no mechanism to force the gradient flow to go through residual block weights during the training. In other words, it might be possible that there are only a few blocks that learn useful representations while a large number of blocks share very little information with small contributions to the ultimate goal. We will show that residual blocks that stay dormant are the chains of blocks at the end of each scale of the residual network.The rest of this paper is organized as follows. Related previous work is reviewed in Sec. 2. Next, we derive a mathematical model for gradient propagation through a layer defined as a combination of batch normalization, convolution layer and ReLU in Sec. 3. Then, we apply this mathematical model to a resnet block in Sec. 4. Afterwards, we use this model to show that the dormant residual blocks are those at the far-end of a scale in deep residual networks in Sec. 5. Concluding remarks and future research directions are given in Sec. 6. We can draw two major conclusions from the analysis conducted above. First, it is proper to relate the above variance analysis to the gradient vanishing and explosion problem. The gradients go through a BN sub-layer in one residual block before moving to the next residual block. As proved in Sec. 3, the gradient mean is zero when it goes through a BN sub-layer and it still stays at zero after passing through a residual block. Thus, if it is normally distributed, the probability of the gradient values between ± 3 standard deviations is 99.7%. A smaller variance would mean lower gradient values. In contrast, a higher variance implies a higher likelihood of discriminatory gradients. Thus, we take the gradient variance across a batch as a measure for stability of gradient backpropagation.Second, recall that the number of filters in each convolution layer of a scale increases by k times with respect to its previous scale. Typically, k = 1 or 2. Without loss of generality, we can assume the following: the variance of weights is about equal across layers, c 1 /c 2 ≈ 1, and k = 2. Then, Eq. (20) can be simplified to DISPLAYFORM0 We see from above that the change in the gradient variance from one residual block to its next is little. This is especially true when the L value is high. This point will be further discussed in the next section.
Batch normalisation maintains gradient variance throughout training, thus stabilizing optimization. This paper analyzed the effect of batch normalization on gradient backpropagation in residual networks