Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned update functions may similarly outperform current hand-designed optimizers, especially for specific tasks. However, learned optimizers are notoriously difficult to train and have yet to demonstrate wall-clock speedups over hand-designed optimizers, and thus are rarely used in practice. Typically, learned optimizers are trained by truncated backpropagation through an unrolled optimization process. The resulting gradients are either strongly biased (for short truncations) or have exploding norm (for long truncations). In this work we propose a training scheme which overcomes both of these difficulties, by dynamically weighting two unbiased gradient estimators for a variational loss on optimizer performance. This allows us to train neural networks to perform optimization faster than well tuned first-order methods. Moreover, by training the optimizer against validation loss, as opposed to training loss, we are able to use it to train models which generalize better than those trained by first order methods. We demonstrate these results on problems where our learned optimizer trains convolutional networks in a fifth of the wall-clock time compared to tuned first-order methods, and with an improvement Gradient based optimization is a cornerstone of modern machine learning. Improvements in optimization have been critical to recent successes on a wide variety of problems. In practice, this typically involves analysis and development of hand-designed optimization algorithms BID22 BID7 BID34 BID15 . These algorithms generally work well on a wide variety of tasks, and are tuned to specific problems via hyperparameter search. On the other hand, a complementary approach is to learn the optimization algorithm BID5 BID30 BID13 BID0 BID36 BID19 BID2 . That is, to learn a function to perform optimization, targeted to particular problems of interest. In this way, the algorithm may learn task specific structure, enabling dramatic performance improvements over more general optimizers.However, training learned optimizers is notoriously difficult. Existing work in this vein can be classified into two broad categories. On one hand are black-box methods such as evolution BID11 BID3 , random search BID6 , reinforcement learning BID2 BID18 BID40 , or Bayesian optimization BID31 . However, these methods scale poorly with the number of optimizer parameters. The other approach is to use first-order methods, by computing the gradient of some measure of optimizer effectiveness with respect to the optimizer parameters. Computing these gradients is costly as we need to iteratively apply the learned update rule, and then backpropagate through these applications, a technique commonly referred to as "unrolled optimization" BID4 BID21 . To address the problem of backpropagation through many optimization steps (analogous to many timesteps in recurrent neural networks), many works make use of truncated backpropagation though time (TBPTT) to partition the long unrolled computational graph into separate pieces BID35 BID33 . This yields computational savings, but at the cost of increased bias BID33 and/or exploding gradients due to many iterated update steps BID26 BID25 . Existing methods address the bias at the cost of increased variance or computa-tional complexity BID38 BID24 BID33 . Previous techniques for training RNNs via TBPTT have thus far not been effective for training optimizers.In this paper, we analytically and experimentally explore the debilitating role of bias and exploding gradients on training optimizers ( §2.3). We then show how these pathologies can be remedied by optimizing the parameters of a distribution over the optimizer parameters, known as variational optimization BID32 ) ( §3). We define two unbiased gradient estimators for this objective: a reparameterization based gradient, and evolutionary strategies BID27 BID23 . By dynamically reweighting the contribution of these two gradient estimators, we are able to avoid strongly biased or exploding gradients, and thus stably and efficiently train learned optimizers.We demonstrate the utility of this approach by training a learned optimizer to target optimization of convolutional networks on image classification ( §4). On the targeted task distribution, this learned optimizer achieves better validation loss, and is five times faster in wall-clock time, compared to well tuned hand-designed optimizers such as SGD+Momentum, RMSProp, and ADAM ( FIG0 ). While not the explicit focus of this work, we also find that the learned optimizer demonstrates promising generalization ability on out of distribution tasks ( Figure 6 ). Training and validation curves for a three layer CNN, trained on a subset of 32x32 imagenet classes not seen during outer-training of the optimizer. Dashed lines indicate the best achieved performance over an additional 130 seconds. We show two learned optimizers -one trained to minimize training loss, and the other trained to minimize validation loss on the inner-problem. We compare against Adam, RMSProp, and SGD+Momentum, individually tuned for the train and validation loss (Panel (a) and (b), respectively). On training loss (a), our learned optimizer approaches zero training loss, and achieves it's smallest loss values in less than one quarter the wall-clock time.On validation loss (b), our learned optimizer achieves a lower minimum, in roughly one third the wall-clock time. Shaded regions correspond to 25 and 75 percentile over five random initializations of the CNN. For plots showing performance in terms of step count rather than wall clock (where we achieve even more dramatic speedups), and for more task instances, see Appendix D. (c) Distribution of the performance difference between the learned optimizers and a tuned baseline of either Adam, RMSProp, or Momentum (loss improvement). Positive values indicate performance better than baseline. We show training and validation losses for the outer-testing task distribution. On the majority of tasks, the learned optimizers outperform the baseline. In this work we demonstrate two difficulties when training learned optimizers: "exploding" gradients, and a bias introduced by truncated backpropagation through time. To combat this, we construct a variational bound of the outer-objective and minimize this via a combination of reparameterization and ES style gradient estimators. By using our combined estimator and a curriculum over truncation step we are able to train learned optimizers that achieve more than five times speedup on wallclock time as compared to existing optimizers.In this work, we focused on applying optimizers to a restricted family of tasks. While useful on its own right (e.g. rapid retraining of models on new data), future work will explore the limits of "no free lunch" BID39 to understand how and when learned optimizers generalize across tasks. We are also interested in using these methods to better understand what problem structure our learned optimizers exploit. By analyzing the trained optimizer, we hope to develop insights that may transfer back to hand-designed optimizers. Outside of meta-learning, we believe the gradient estimator presented here can be used to train other long time dependence recurrent problems such as neural turning machines BID12 , or neural GPUs BID14 .Much in the same way deep learning has replaced feature design for perceptual tasks, we see metalearning as a tool capable of learning new and interesting algorithms, especially for domains with unexploited problem-specific structure. With better outer-training stability, we hope to improve our ability to learn interesting algorithms, both for optimizers and beyond.
We analyze problems when training learned optimizers, address those problems via variational optimization using two complementary gradient estimators, and train optimizers that are 5x faster in wall-clock time than baseline optimizers (e.g. Adam). This paper uses un-rolled optimization to learn neural networks for optimization. This paper tackles the problem of learning an optimizer, specifically the authors focus on obtaining cleaner gradients from the unrolled training procedure. Presents a method for "learning an optimizer" by using a Variational Optimization for the "outer" optimizer loss and proposes the idea to combine both the reparametrized gradient and the score-function estimator for the Variational Objective and weights them using a product of Gaussians formula for the mean.