Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation. Artificial intelligence applications have witnessed major advances in recent years. At the core of this revolution is the development of novel neural network models and their training techniques. For example, since the landmark work of BID13 , most of the state-of-the-art image recognition systems are built upon a deep stack of network blocks consisting of convolutional layers and additive skip connections, with some normalization mechanism (e.g., batch normalization BID16 ) to facilitate training and generalization. Besides image classification, various normalization techniques (Ulyanov et al., 2016; BID0 BID26 Wu & He, 2018) have been found essential to achieving good performance on other tasks, such as machine translation (Vaswani et al., 2017) and generative modeling (Zhu et al., 2017) . They are widely believed to have multiple benefits for training very deep neural networks, including stabilizing learning, enabling higher learning rate, accelerating convergence, and improving generalization.• Training without normalization. We propose Fixup, a method that rescales the standard initialization of residual branches by adjusting for the network architecture. Fixup enables training very deep residual networks stably at maximal learning rate without normalization. In the remaining of this paper, we first analyze the exploding gradient problem of residual networks at initialization in Section 2. To solve this problem, we develop Fixup in Section 3. In Section 4 we quantify the properties of Fixup and compare it against state-of-the-art normalization methods on real world benchmarks. A comparison with related work is presented in Section 5. In this work, we study how to train a deep residual network reliably without normalization. Our theory in Section 2 suggests that the exploding gradient problem at initialization in a positively homogeneous network such as ResNet is directly linked to the blowup of logits. In Section 3 we develop Fixup initialization to ensure the whole network as well as each residual branch gets updates of proper scale, based on a top-down analysis. Extensive experiments on real world datasets demonstrate that Fixup matches normalization techniques in training deep residual networks, and achieves state-of-the-art test performance with proper regularization.Our work opens up new possibilities for both theory and applications. Can we analyze the training dynamics of Fixup, which may potentially be simpler than analyzing models with batch normalization is? Could we apply or extend the initialization scheme to other applications of deep learning? It would also be very interesting to understand the regularization benefits of various normalization methods, and to develop better regularizers to further improve the test performance of Fixup. Proof of Theorem 1. We use f i→j to denote the composition DISPLAYFORM0 . Note that z is p.h. with respect to the input of each network block, i.e. f i→L ((1 + )x i−1 ) = (1 + )f i→L (x i−1 ) for > −1. This allows us to compute the gradient of the cross-entropy loss with respect to the scaling factor at = 0 as DISPLAYFORM1 Since the gradient L 2 norm ∂ /∂xi−1 must be greater than the directional derivative DISPLAYFORM2 xi−1 ), y), defining = t / xi−1 we have DISPLAYFORM3
All you need to train deep residual networks is a good initialization; normalization layers are not necessary. A method is presented for initialization and normalization of deep residual networks. This is based on observations of forward and backward explosion in such networks. The method performance is on par with the best results obtained by other networks with more explicit normalization. The authors propose a novel way to initialize residual networks, which is motivated by the need to avoid exploding/vanishing gradients. Proposes a new initialization method used to train very deep RedNets without using batch-norm.