Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-the-art result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. Code is immediately available. Stochastic gradient descent (SGD) serves as the optimizer of choice for many recent advances in deep learning across domains (Krizhevsky et al., 2012; He et al., 2016a; . SGD for deep learning is typically augmented with either the "heavy ball" momentum technique of Polyak (1964) or the accelerated gradient of Nesterov (1983) . In the deterministic setting, these methods provably yield faster convergence in fairly general settings. In the stochastic setting, these methods lose many theoretical advantages. However, due to its implicit gradient averaging, momentum can confer the benefit of variance reduction, applying less noisy parameter updates than plain SGD. Recent work has explicitly shown the use of momentum as a variance reducer (Roux et al., 2018) .Algorithms Starting with gradient variance reduction as an informal and speculative motivation, we introduce the quasi-hyperbolic momentum (QHM) optimization algorithm in Section 3. Put as simply as possible, QHM's update rule is a weighted average of momentum's and plain SGD's update rule. We later propose a similar variant of Adam (QHAdam) in Section 5.Connecting the dots QHM is simple yet expressive. In Section 4, we connect QHM with plain SGD, momentum, Nesterov's accelerated gradient, PID control algorithms (Recht, 2018; , synthesized Nesterov variants (Lessard et al., 2016) , noise-robust momentum (Cyrus et al., 2018) , Triple Momentum (Scoy et al., 2018) , and least-squares acceleration of SGD (Kidambi et al., 2018) . Such connections yield reciprocal benefits -these algorithms aid in analyzing QHM, and conversely QHM recovers many of these algorithms in a more efficient and conceptually simpler manner. We then characterize the set of optimization algorithms that QHM recovers. Theoretical convergence results We note that various convergence results follow simply via these connections. In the deterministic (full-batch) case, since QHM recovers Triple Momentum, QHM also recovers the global linear convergence rate of 1 − 1/ √ κ for strongly convex, smooth loss functions.6 For first-order methods, this is the fastest known global convergence rate for such functions. In the stochastic (minibatch) case, QHM's recovery of AccSGD gives QHM the same convergence results as in Kidambi et al. (2018) 's least-squares regression setting, of O( √ κ · log κ · log 1 ) iterations for -approximation of the minimal loss.Unifying two-state optimization algorithms These connections demonstrate that many two-state optimization algorithms are functionally similar or equivalent to each other. However, they are often implemented inefficiently and their parameterizations can be inaccessible to practitioners. QHM yields a highly accessible and efficient version of these algorithms. Polyak, 1964) subfamily better recovered by QHM with ν = 1 NAG (Nesterov, 1983) subfamily same recovered by QHM with ν = β PID (Recht, 2018) parent worse QHM's β restricts PID's k P /k D PID bijective worse degenerate; either "PI" or "PD" SNV (Lessard et al., 2016) bijective worse used in handling multiplicative noise Robust M. (Cyrus et al., 2018) subfamily worse SNV w/ convergence guarantees Triple M. (Scoy et al., 2018) subfamily worse "fastest" for str. convex, smooth L(·) AccSGD (Kidambi et al., 2018) subfamily worse acceleration for least-squares SGD * "subfamily" means that QHM recovers the algorithm but not vice-versa. "parent" means that the algorithm recovers QHM but not vice-versa. "bijective" means that the algorithms recover each other. † Efficiency (compute and/or memory) vs. QHM.In Appendix D, we characterize the set of two-state optimization algorithms recoverable by QHM. Our hope here is to provide future work with a routine conversion to QHM so that they may leverage the accessibility and efficiency benefits, as well as the many connections to other algorithms.Many-state optimization algorithms Going beyond a single momentum buffer, it is possible to recover many-state algorithms by linearly combining many momentum buffers (with different discount factors) in the update rule. However, we found in preliminary experiments that using multiple momentum buffers yields negligible value over using a single slow-decaying momentum buffer and setting an appropriate immediate discount -that is, using QHM with high β and appropriate ν.We note that the Aggregated Momentum (AggMo) algorithm (Lucas et al., 2018) precisely performs this linear combination of multiple momentum buffers. While AggMo takes a simple average of the buffers, an extended variant of AggMo allows for other linear combinations. This extended AggMo can be viewed as a many-state generalization of two-state algorithms (including QHM), recovering them when two buffers are used. Appendix H provides a supplemental discussion and empirical comparison of QHM and AggMo, corroborating our preliminary experiments' findings. QHM and QHAdam are computationally cheap, intuitive to interpret, and simple to implement. They can serve as excellent replacements for momentum/NAG and Adam in a variety of settings. In particular, they enable the use of high exponential discount factors (i.e. β) through the use of immediate discounting (i.e. ν). QHM recovers numerous other algorithms in an efficient and accessible manner. Parameter sweep experiments and case studies demonstrate that the QH algorithms can handily outpace their vanilla counterparts. We hope that practitioners and researchers will find these algorithms both practically useful and interesting as a subject of further study. The recommended vanilla Adam setting of β 2 = 0.999 in Kingma & Ba (2015) makes the right-hand side of (19) to be large, and various work has employed Adam with a significantly lower β 2 ; e.g. 0.98 BID12 BID15 . 26 Decreasing β 2 is undesirable, often slowing down training. 27 Moving from Adam to QHAdam, an alternative solution is to decrease ν 2 to be below 1. This decreases the right-hand side of (18), up to a point, and thus imposes a tighter constraint on the magnitudes of updates than the vanilla Adam setting of ν 2 = 1. Fig. 3 shows an example of this phenomenon using a fixed ν 1 , β 1 , and β 2 .Figure 3: Bound from (18), fixing ν 1 = 0.8, β 1 = 0.95, and β 2 = 0.98, and varying ν 2 . 26 We performed experiments on these models indicating that increasing β2 far beyond 0.98 led to training explosion. We suspect that these instability issues are especially prevalent in settings with rare inputs or labels, such as machine translation. 27 In proposing the AdamNC algorithm, Reddi et al. (2018) suggests that β2 should be high to capture a sufficiently long history of past gradients. To recap, we take the optimal AggMo parameterization from an extensive sweep, we convert that parameterization by hand to one for QHM, and we find that the latter outperforms the former on this autoencoder task.These results indicate that using multiple momentum buffers with an arbitrary weighting scheme (i.e. AggMo with K > 2) provides negligible benefit over using a single slow-decaying momentum buffer with an appropriate weight (i.e. QHM with high β and appropriate ν). Lucas et al. (2018) offer an interpretation of AggMo as passive damping for physical systems. In this interpretation, fast-decaying momentum buffers "dampen" the oscillations of slow-decaying momentum buffers by providing velocity in an opposite direction.
Mix plain SGD and momentum (or do something similar with Adam) for great profit. The paper proposes simple modifications to SGD and Adam, called QH-variants, that can recover the “parent” method and a host of other optimization tricks. A variant of classical momentum which takes a weighted average of momentum and gradient update, and an evaluation of its relationships between other momentum based optimization schemes.