In this paper, we show a phenomenon, which we named ``super-convergence'', where residual networks can be trained using an order of magnitude fewer iterations than is used with standard training methods.    The existence of super-convergence is relevant to understanding why deep networks generalize well.   One of the key elements of super-convergence is training with cyclical learning rates and a large maximum learning rate.   Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network. In addition, we show that super-convergence provides a  greater boost in performance relative to standard training when the amount of labeled training data is limited.   We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate.   The architectures to replicate this work will be made available upon publication.
 While deep neural networks have achieved amazing successes in a range of applications, understanding why stochastic gradient descent (SGD) works so well remains an open and active area of research. This paper provides unique empirical evidence that supports the theories in some papers but not others. Specifically, we show that, for certain datasets, residual network architectures BID15 , and hyper-parameter values, using very large learning rates with the cyclical learning rate (CLR) method BID29 BID30 can speed up training by an order of magnitude. Analogous to the phenomenon of super-conductivity that only happens in limited circumstances and provides theoretical insights of materials, we named this phenomenon "super-convergence." While super-convergence might be of some practical value, the primary purpose of this paper is to provide empirical support and theoretical insights to the active discussions in the literature on SGD and understanding generalization. FIG0 provides a comparison of test accuracies from a super-convergence example and the result of a typical (piecewise constant) training regime for Cifar-10, both using a 56 layer residual network architecture. Piecewise constant training reaches a peak accuracy of 91.2% after approximately 80,000 iterations, while the super-convergence method reaches a higher accuracy (92.4%) after only 10,000 iterations. FIG1 shows the results for a range of CLR stepsize values, where training lasted only one cycle. This modified learning rate schedule achieves a higher final test accuracy (92.1%) than typical training (91.2%) after only 6,000 iterations. In addition, as the total number of iterations increases from 2,000 to 20,000, the final accuracy improves from 89.7% to 92.7%.The contributions of this paper are:1. We demonstrate a new training phenomenon and systematically investigate it. 2. We show evidence that large learning rates regularize the trained network and hypothesize that this allows SGD to find a flat local minima that generalizes well. 3. We derive a simplification of the second order, Hessian-free optimization method to estimate optimal learning rates which demonstrates that large learning rates find wide, flat minima. 4. We demonstrate that the effects of super-convergence are increasingly dramatic when less labeled training data is available. There is substantial discussion in the literature on stochastic gradient descent (SGD) and understanding why solutions generalize so well (i.e, Chaudhari et al. FORMULA1 BID20 ). Super-convergence provides empirical evidence that supports some theories, contradicts some others, and points to the need for further theoretical understanding. We hope the response to super-convergence is similar to the reaction to the initial report of network memorization , which sparked an active discussion within the deep learning research community on better understanding of the factors in SGD leading to solutions that generalize well (i.e., ).Our work impacts the line of research on SGD and the importance of noise for generalization. In this paper we focused on the use of CLR with very large learning rates, which adds noise in the middle part of training. Recently , stated that higher levels of noise lead SGD to solutions with better generalization. Specifically , they showed that the ratio of the learning rate to the batch size, along with the variance of the gradients, controlled the width of the local minima found by SGD. Independently , BID6 show that SGD performs regularization by causing SGD to be out of equilibrium, which is crucial to obtain good generalization performance, and derive that the ratio of the learning rate to batch size alone controls the entropic regularization term. They also state that data augmentation increases the diversity of SGD's gradients, leading to better generalization. These two papers provide theoretical support for the super-convergence phenomenon. In addition there are several other papers in the literature which state that wide, flat local minima produce solutions that generalize better than sharp minima BID16 BID21 BID36 . Our super-convergence results align with these results.In addition, there are several recent papers on the generalization gap between small and large minibatches and the relationship between gradient noise, learning rate, and batch size. Our results here supplements this other work by illustrating the possibility of time varying high noise levels during training. As mentioned above, showed that SGD noise is proportional to the learning rate, the variance of the loss gradients, divided by the batch size. Similarly Smith and Le derived the noise scale as g ≈ N/B(1 − m), where g is the gradient noise, the learning rate, N the number of training samples, and m is the momentum coefficient. Furthermore, showed an equivalence of increasing batch sizes instead of a decreasing learning rate schedule. Importantly, these authors demonstrated that the noise scale g is relevant to training and not the learning rate or batch size. BID21 study the generalization gap between small and large mini-batches, stating that small mini-batch sizes lead to wide, flat minima and large batch sizes lead to sharp minima. They also suggest a batch size warm start for the first few epochs, then using a large batch size, which amounts to training with large gradient noise for a few epochs and then removing it. Our results contradicts this suggestion as we found it preferable to start training with little or no noise and let it increase (i.e., curriculum training), reach a noise peak, and reduce the noise level in the last part of training (i.e., simulated annealing). BID12 use a very large mini-batch size of up to 8,192 and adjust the learning rate linearly with the batch size. They also suggest a gradual warmup of the learning rate, which is a discretized version of CLR and matches our experience with an increasing learning rate. They make a point relevant to adjusting the batch size; if the network uses batch normalization, different mini-batch sizes leads to different statistics, which must be handled. BID17 made a similar point about batch norm and suggested using ghost statistics. Also, BID17 show that longer training lengths is a form of regularization that improves generalization. On the other hand, our results show a different form of regularization that comes from training with very large learning rates, which permits much shorter training lengths.Furthermore, this paper points the way towards new research directions, such as the following three:1. Characterizing "good noise" that improves the trained network's ability to generalize versus "bad noise" that interferes with finding a good solution (i.e., ). We find that there is a lack of a unified framework for treating SGD noise/diversity, such as architectural noise (e.g., dropout BID33 , dropconnect BID35 ), noise from hyper-parameter settings (e.g., learning rate, mini-batch size), adding gradient noise, adding noise to weights BID9 , and input diversity (e.g., data augmentation, noise). Gradient diversity has been shown to lead to flatter local minimum and better generalization. A unified framework should resolve conflicting claims in the literature on the value of each of these, such as for architectural noise BID33 ) versus Hoffer et al. (2017 ). Furthermore, many papers study each of these factors independently and by focusing on the trees, one might miss the forest. 2. Time dependent application of good noise during training: As described above, combining curriculum learning with simulated annealing leads to cyclical application. To the best of our knowledge, this has only been applied sporadically in a few methods such as CLR BID30 or cyclical batch sizes but these all fall under a single umbrella of time dependent gradient diversity (i.e., noise). Also, one might learn an optimal noise level while training the network. 3. Discovering new ways to stabilize optimization (i.e., SGD) with large noise levels: Our evidence indicates that normalization (and batch normalization in particular) is the catalyst enabling super-convergence in the face of destabilizing noise from the large learning rates. Normalization methods (batch norm, layer normalization BID2 , streaming normalization BID23 ) and new techniques (i.e., cyclical gradient clipping) to stabilize training need further investigation to discover better ways to keep SGD stable in the presence of enormous good noise.Classical physics was insufficient for explaining super-conductivity when it was discovered and pointed to the need for new theories, such as quantum mechanics. Similarly, super-convergence indicates a need for new theories of SGD and generalization.The Resnet-56 architecture used consists of three stages. Within each stage, the same residual block structure is sequentially repeated . This structure is given in TAB1 . Between stages, a different residual block structure is used to reduce the spatial dimension of the channels. TAB2 shows this structure. The overall architecture is described in TAB3 . Following the Caffe convention, each Batch Norm layer was followed by a scaling layer to achieve true batch normalization behavior. This and the other architectures necessary to replicate this work will be made available upon publication. We ran a wide variety of experiments and due to space limitations in the main article, we report some of the more interesting results here that did not fit in the main article.In the main text we only showed the results of super-convergence for the Cifar-10 dataset. In fact, the super-convergence phenomenon also occurs with Cifar-100, which implies an independence of this phenomenon on the number of classes. FIG0 shows the results of the LR range test for Resnet-56 with the Cifar-100 training data. The curve is smooth and accuracy remains high over the entire range from 0 to 3 indicating a potential for super-convergence. An example of super-convergence with Cifar-100 with Resnet-56 is given in FIG16 , where there is also a comparison to the results from a piecewise constant training regime. Furthermore, the final accuracy for the super-convergence curve is 68.6%, while the accuracy for the piecewise constant method is 59.8%, which is an 8.8% improvement.As discussed in the main text, we tested various adaptive learning rate methods with Resnet-56 training on Cifar-10 to determine if they are capable of recognizing the need for using very large learning rates. FIG0 shows the results of this training for Nesterov momentum BID34 BID27 , AdaDelta BID8 ), AdaGrad (Zeiler, 2012 , and Adam (Kingma BID22 . We found no sign that any of these methods discovered the utility of large learning rates nor any indication of super-convergence-like behavior. We also ran CLR with these adaptive learning methods and found that Nesterov, AdaDelta, and AdaGrad allowed super-convergence to occur, but we were unable to create this phenomenon with Adam. For example, FIG18 shows a comparison of super-convergence to a piecewise constant training regime with the Nesterov momentum method. Here super-convergence yields a final test accuracy after 10,000 iterations of 92.1% while the piecewise constant training regime at iteration 80,000 has an accuracy of 90.9%. FIG1 shows a comparison of runs of the super-convergence phenomenon, both with and without dropout. The LR range test with and without dropout is shown in FIG0 . FIG1 shows the results of training for 10,000 iterations . In both cases, the dropout ratio was set to 0.2 and the Figure shows a small improvement with dropout. We also ran with other values for the dropout ratio and consistently saw similar improvements.The effect of mini-batch size is discussed in the main paper but here we present a table containing the final accuracies of super-convergence training with various mini-batch sizes. One can see in Table 5 the final test accuracy results and this table shows that the larger the mini-batch size, the better the final accuracy, which differs from the results shown in the literature 4 . Most of results reported in this paper are with a total mini-batch size of 1,000.In addition, we ran experiments on Resnet-56 on Cifar-10 with modified values for momentum and weight decay to determine if they might hinder the super-convergence phenomenon. FIG0 shows the results for momentum set to 0.8, 0.85, 0.9, and 0.95 and the final test accuracies are listed in TAB5 . These results indicate only a small change in the results, with a setting of 0.9 being a bit better than the other values. In FIG1 are the results for weight decay values of 10 −3 , 10 −4 , 10 −5 , and 10 −6 . In this case, a weight decay value of 10 −3 prevents super-convergence, while the smaller values do not. This FIG0 shows that a weight decay value of 10 −4 performs well. Table 5 : Comparison of accuracy results for various total training batch sizes with Resnet-56 on Cifar-10 using CLR=0.1-3 and stepsize=5,000.
Empirical proof of a new phenomenon requires new theoretical insights and is relevent to the active discussions in the literature on SGD and understanding generalization. The paper discusses a phenomenon where neural network training in very specific settings can profit much from a schedule including large learning rates The authors analyze training of residual networks using large cyclic learning rates, and demonstrate fast convergence with cyclic learning rates and evidence of large learning rates acting as regularization.