Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such out-of-equilibrium behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix. Our first result is to show precisely in what sense stochastic gradient descent (SGD) implicitly performs variational inference, as is often claimed informally in the literature. For a loss function f (x) with weights x ∈ R d , if ρ ss is the steady-state distribution over the weights estimated by SGD, DISPLAYFORM0 where H(ρ) is the entropy of the distribution ρ and η and b are the learning rate and batch-size, respectively. The potential Φ(x), which we characterize explicitly, is related but not necessarily equal to f (x). It is only a function of the architecture and the dataset. This implies that SGD implicitly performs variational inference with a uniform prior, albeit of a different loss than the one used to compute back-propagation gradients.We next prove that the implicit potential Φ(x) is equal to our chosen loss f (x) if and only if the noise in mini-batch gradients is isotropic. This condition, however, is not satisfied for deep networks. Empirically, we find gradient noise to be highly non-isotropic with the rank of its covariance matrix being about 1% of its dimension. Thus, SGD on deep networks implicitly discovers locations where ∇Φ(x) = 0, these are not the locations where ∇ f (x) = 0. This is our second main result: the most likely locations of SGD are not the local minima, nor the saddle points, of the original loss. The deviation of these critical points, which we compute explicitly scales linearly with η/b and is typically large in practice.When mini-batch noise is non-isotropic, SGD does not even converge in the classical sense. We prove that, instead of undergoing Brownian motion in the vicinity of a critical point, trajectories have a deterministic component that causes SGD to traverse closed loops in the weight space. We detect such loops using a Fourier analysis of SGD trajectories. We also show through an example that SGD with non-isotropic noise can even converge to stable limit cycles around saddle points. The continuous-time point-of-view used in this paper gives access to general principles that govern SGD, such analyses are increasingly becoming popular BID61 BID9 . However, in practice, deep networks are trained for only a few epochs with discrete-time updates. Closing this gap is an important future direction. A promising avenue towards this is that for typical conditions in practice such as small mini-batches or large learning rates, SGD converges to the steady-state distribution quickly BID48 . Let F(ρ) be as defined in (11). In non-equilibrium thermodynamics, it is assumed that the local entropy production is a product of the force −∇ δ F δ ρ from (A8) and the probability current −J(x,t) from (FP). This assumption in this form was first introduced by BID46 based on the works of BID41 BID40 . See Frank (2005, Sec. 4 .5) for a mathematical treatment and Jaynes (1980) for further discussion. The rate of entropy (S i ) increase is given by DISPLAYFORM0 This can now be written using (A8) again as DISPLAYFORM1 The first term in the above expression is non-negative, in order to ensure that DISPLAYFORM2 where the second equality again follows by integration by parts. It can be shown (Frank, 2005, Sec. 4.5.5 ) that the condition in Assumption 4, viz., ∇ · j(x) = 0, is sufficient to make the above integral vanish and therefore for the entropy generation to be non-negative.C SOME PROPERTIES OF THE FORCE jThe Fokker-Planck equation (FP) can be written in terms of the probability current as DISPLAYFORM3 Since we have ρ ss ∝ e −β Φ(x) , from the observation (7), we also have that DISPLAYFORM4 and consequently, DISPLAYFORM5 In other words, the conservative force is non-zero only if detailed balance is broken, i.e., J ss = 0. We also have DISPLAYFORM6 which shows using Assumption 4 and ρ ss (x) > 0 for all x ∈ Ω that j(x) is always orthogonal to the gradient of the potential DISPLAYFORM7 Using the definition of j(x) in (8), we have detailed balance when DISPLAYFORM8
SGD implicitly performs variational inference; gradient noise is highly non-isotropic, so SGD does not even converge to critical points of the original loss This paper provides a variational analysis of SGD as a non-equilibrium process. This paper discusses the regularized objective function minimized by standard SGD in the context of neural nets, and provide a variational inference perspective using the Fokker-Planck equation. Develops a theory to study the impact of stocastic gradient noise for SGD, especially for deep neural network models