Kingma et al. introduced a variant of this algorithm that adds estimates of biased and corrected first and second moments. Schaul et al. introduced an adaptive variant of it that requires a good estim
Kingma et al. introduced a variant of this algorithm that adds estimates of biased and corrected first and second moments. Schaul et al. introduced an adaptive variant of it that requires a good estimate of the Hessian, while SMORMS3 is a simpler hybrid of that variant and RMSProp, a version of this algorithm introduced in an oft- cited lecture slide by Geoffrey Hinton. Hogwild! is a lock-free implementation of an asynchronous version of this algorithm, while Nesterov and Rumelhart et al. each introduced variants that use momentum. (*) Stochastic approaches to this algorithm perform each step with a small sample or minibatch from a dataset and form the basis for optimization of deep neural networks. For 10 points, Adam and AdaGrad are variants of what simplest first -order optimization algorithm, which finds a local minimum for a function by taking steps proportional to and in the direction of highest negative slope?
Kingma et al. introduced a variant of this algorithm that adds estimates of biased and corrected first and second moments.
gradient descent [or  steepest descent, accept  SGD]