The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn’t. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of the stochastic gradient, whereas the update magnitude is solely determined by an estimate of its relative variance. We  disentangle these two aspects and analyze them in isolation, shedding light on ADAM ’s inner workings. Transferring the "variance adaptation” to momentum- SGD gives rise to a novel method, completing the practitioner’s toolbox for problems where ADAM fails. Many prominent machine learning models pose empirical risk minimization problems of the form DISPLAYFORM0 where θ ∈ R d is a vector of parameters, {x 1 , . . . , x M } a training set, and (θ; x) is a loss quantifying the performance of parameter vector θ on example x. Computing the exact gradient in each step of an iterative optimization algorithm becomes inefficient for large M . Instead, we construct a minibatch B ⊂ {1, . . . , M } of |B| M data points sampled uniformly and independently from the training set and compute an approximate stochastic gradient DISPLAYFORM1 which is an unbiased estimate, E[g(θ)] = ∇L(θ). We will denote by σ(θ) The basic stochastic optimizer is stochastic gradient descent (SGD, Robbins & Monro, 1951) and its momentum variants (Polyak, 1964; Nesterov, 1983) . A number of methods, widely-used in the deep learning community, choose per-element update magnitudes based on the history of stochastic gradient observations. Among these are ADAGRAD (Duchi et al., 2011) , RMSPROP (Tieleman & Hinton, 2012) , ADADELTA (Zeiler, 2012) and ADAM (Kingma & Ba, 2015) . We have argued that ADAM combines two aspects: taking signs and variance adaptation. Our separate analysis of both aspects provides some insight into the inner workings of this method. Taking the sign can be beneficial, but does not need to be. Our theoretical analysis suggests that it depends on the interplay of stochasticity, the conditioning of the problem, and its "axis-alignment". Our experiments confirm that sign-based methods work well on some, but not all problems.Variance adaptation can be applied to any stochastic update direction. In our experiments it was beneficial in all cases, but its effect can sometimes be minuscule. M-SVAG, a variance-adapted variant of momentum-SGD, is a useful addition to the practitioner's toolbox for problems where sign-based methods like ADAM fail. Its memory and computation cost are identical to ADAM and it has two hyper-parameters, the momentum constant µ and the global step size α. Our TensorFlow (Abadi et al., 2015) implementation of this method will be made available upon publication. MNIST We train a simple fully-connected neural network with three hidden layers of 1000, 500 and 100 units with ReLU activation. The output layer has 10 units with softmax activation. We use the cross-entropy loss function and apply L 2 -regularization on all weights, but not the biases. We use a batch size of 128. The global learning rate α stays constant.
Analyzing the popular Adam optimizer The paper trys to improve Adam based on variance adaption with momentum by proposing two algorithms This paper analyzes the scale-invariance and the particular shape of the learning rate used in Adam, arguing that Adam's update is a combination of a sign-update and a variance-based learning rate. The paper splits ADAM algorithm into two components: stochastic direction in sign of gradient and adaptive stepwise with relative variance, and two algorithms are proposed to test each of them.