The application of stochastic variance reduction to optimization has shown remarkable recent theoretical and practical success. The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem. We show that naive application of the SVRG technique and related approaches fail, and explore why. Stochastic variance reduction (SVR) consists of a collection of techniques for the minimization of finite-sum problems: DISPLAYFORM0 such as those encountered in empirical risk minimization, where each f i is the loss on a single training data point. Principle techniques include SVRG BID15 , SAGA BID7 , and their variants. SVR methods use control variates to reduce the variance of the traditional stochastic gradient descent (SGD) estimate f i (w) of the full gradient f (w). Control variates are a classical technique for reducing the variance of a stochastic quantity without introducing bias. Say we have some random variable X. Although we could use X as an estimate of E[X] =X, we can often do better through the use of a control variate Y . If Y is a random variable correlated with X (i.e. Cov[X, Y ] > 0), then we can estimateX with the quantity Remarkably, these methods are able to achieve linear convergence rates for smooth strongly-convex optimization problems, a significant improvement on the sub-linear rate of SGD. SVR methods are part of a larger class of methods that explicitly exploit finite-sum structures, either by dual (SDCA, BID25 MISO, Mairal, 2014; Finito, Defazio et al., 2014b) or primal (SAG, Schmidt et al., 2017) approaches. DISPLAYFORM1 Recent work has seen the fusion of acceleration with variance reduction BID26 ; BID21 ; BID6 ; BID1 ), and the extension of SVR approaches to general non-convex BID2 BID23 as well as saddle point problems BID3 .In this work we study the behavior of variance reduction methods on a prototypical non-convex problem in machine learning: A deep convolutional neural network designed for image classification. We discuss in Section 2 how standard training and modeling techniques significantly complicate the application of variance reduction methods in practice, and how to overcome some of these issues. In Sections 3 & 5 we study empirically the amount of variance reduction seen in practice on modern CNN architectures, and we quantify the properties of the network that affect the amount of variance reduction. In Sections 6 & 7 we show that streaming variants of SVRG do not improve over regular SVRG despite their theoretical ability to handle data augmentation. In Section 8 we study properties of DNN problems that actually give stochastic gradient descent an advantage over variance reduction techniques. The negative results presented here are disheartening, however we don't believe that they rule out the use of stochastic variance reduction on deep learning problems. Rather, they suggest avenues for further research. For instance, SVR can be applied adaptively; or on a meta level to learning rates; or scaling matrices; and can potentially be combined with methods like Adagrad BID9 and ADAM BID17 to yield hybrid methods.
The SVRG method fails on modern deep learning problems This paper presents an analysis of SVRG style methods, showing that dropout, batch norm, data augmentation (random crop/rotation/translations) tend to increase bias and/or variance of the updates. This paper investigates the applicability of SVGD to modern neural networks and shows the naive application of SVGD typically fails.