Subhransu Maji, and Jitendra Malik. Semantic contours from
inverse detectors. In International Conference on Computer
Vision (ICCV) , 2011. 6
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In The IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , June 2016. 4
[12] Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled deep neural network for semi-supervised semantic
segmentation. In NIPS) , December 2015. 1, 2
[13] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey
Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama,
and Trevor Darrell. Caffe: Convolutional architecture for fast
feature embedding. arXiv preprint arXiv:1408.5093 , 2014.
2, 6
[14] Iasonas Kokkinos. Surpassing humans in boundary detection
using deep learning. CoRR , abs/1511.07386, 2015. 2[15] Philipp Krähenbühl and Vladlen Koltun. Efﬁcient inference
in fully connected crfs with gaussian edge potentials. In


in fully connected crfs with gaussian edge potentials. In
J. Shawe-Taylor, R.S. Zemel, P.L. Bartlett, F. Pereira, and
K.Q. Weinberger, editors, Advances in Neural Information
Processing Systems 24 , pages 109–117. Curran Associates,
Inc., 2011. 1, 7
[16] John D. Lafferty, Andrew McCallum, and Fernando C. N.
Pereira. Conditional random ﬁelds: Probabilistic models for
segmenting and labeling sequence data. In Proceedings of
the Eighteenth International Conference on Machine Learning, ICML ’01, pages 282–289, San Francisco, CA, USA,
2001. Morgan Kaufmann Publishers Inc. 3
[17] Guosheng Lin, Chunhua Shen, Ian D. Reid, and Anton
van den Hengel. Efﬁcient piecewise training of deep
structured models for semantic segmentation. CoRR ,
abs/1504.01013, 2015. 1, 2, 3
[18] Ce Liu, Jenny Yuen, and Antonio Torralba. Nonparametric
Scene Parsing via Label Transfer , pages 207–236. Springer
International Publishing, Cham, 2016. 6, 7

Convolutional Random Walk Networks for Semantic Image Segmentation

Summary:
Most current semantic segmentation methods rely on fully convolutional
networks (FCNs). However, their use of large receptive fields and many pooling
layers cause low spatial resolution inside the deep layers. This leads to
predictions with poor localization around the boundaries. Prior work has
attempted to address this issue by post-processing predictions with CRFs or
MRFs. But such models often fail to capture semantic relationships between
objects, which causes spatially disjoint predictions. To overcome these
problems, recent methods integrated CRFs or MRFs into an FCN framework. The
downside of these new models is that they have much higher complexity than
traditional FCNs, which renders training and testing more challenging.
  In this work we introduce a simple, yet effective Convolutional Random Walk
Network (RWN) that addresses the issues of poor boundary localization and
spatially fragmented predictions with very little increase in model complexity.
Our proposed RWN jointly optimizes the objectives of pixelwise affinity and
semantic segmentation. It combines these two objectives via a novel random walk
layer that enforces consistent spatial grouping in the deep layers of the
network. Our RWN is implemented using standard convolution and matrix
multiplication. This allows an easy integration into existing FCN frameworks
and it enables end-to-end training of the whole network via standard
back-propagation. Our implementation of RWN requires just $131$ additional
parameters compared to the traditional FCNs, and yet it consistently produces
an improvement over the FCNs on semantic segmentation and scene labeling.

Gedas Bertasius
Lorenzo Torresani
Stella X. Yu
Jianbo Shi


Scene Parsing via Label Transfer , pages 207–236. Springer
International Publishing, Cham, 2016. 6, 7
[19] Ziwei Liu, Xiaoxiao Li, Ping Luo, Chen Change Loy, and
Xiaoou Tang. Semantic image segmentation via deep parsing
network. In ICCV , 2015. 1, 2, 3
[20] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
convolutional networks for semantic segmentation. In The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015. 1, 2
[21] Laszlo Lovasz. Random walks on graphs: A survey, 1993.
2, 3
[22] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database
of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 7
[23] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han.
Learning deconvolution network for semantic segmentation.
InComputer Vision (ICCV), 2015 IEEE International Conference on , 2015. 1, 2
[24] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry


InComputer Vision (ICCV), 2015 IEEE International Conference on , 2015. 1, 2
[24] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry
Winograd. The pagerank citation ranking: Bringing order
to the web. Technical Report 1999-66, Stanford InfoLab,
November 1999. Previous number = SIDL-WP-1999-0120.
2, 3
[25] Xiaojuan Qi, Jianping Shi, Shu Liu, Renjie Liao, and Jiaya Jia. Semantic segmentation with object clique potential.
InThe IEEE International Conference on Computer Vision
(ICCV) , December 2015. 2
[26] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR ,
abs/1409.1556, 2014. 4
[27] Shuai Zheng, Sadeep Jayasumana, Bernardino RomeraParedes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang
Huang, and Philip Torr. Conditional random ﬁelds as recurrent neural networks. In International Conference on Computer Vision (ICCV) , 2015. 1, 2, 3


Learning to learn by gradient descent
by gradient descent
Marcin Andrychowicz1, Misha Denil1, Sergio Gómez Colmenarejo1, Matthew W. Hoffman1,
David Pfau1, Tom Schaul1, Brendan Shillingford1;2, Nando de Freitas1;2;3
1Google DeepMind2University of Oxford3Canadian Institute for Advanced Research
marcin.andrychowicz@gmail.com
{mdenil,sergomez,mwhoffman,pfau,schaul}@google.com
brendan.shillingford@cs.ox.ac.uk ,nandodefreitas@google.com
Abstract
The move from hand-designed features to learned features in machine learning has
been wildly successful. In spite of this, optimization algorithms are still designed
by hand. In this paper we show how the design of an optimization algorithm can be
cast as a learning problem, allowing the algorithm to learn to exploit structure in
the problems of interest in an automatic way. Our learned algorithms, implemented
by LSTMs, outperform generic, hand-designed competitors on the tasks for which
they are trained, and also generalize well to new tasks with similar structure. We
demonstrate this on a number of tasks, including simple convex problems, training
neural networks, and styling images with neural art.

Marcin Andrychowicz
Misha Denil
Sergio Gomez
Matthew W. Hoffman
David Pfau
Tom Schaul
Brendan Shillingford
Nando de Freitas


demonstrate this on a number of tasks, including simple convex problems, training
neural networks, and styling images with neural art.
1 Introduction
Frequently, tasks in machine learning can be expressed as the problem of optimizing an objective
functionf()deﬁned over some domain 2. The goal in this case is to ﬁnd the minimizer
= arg min2f(). While any method capable of minimizing this objective function can be
applied, the standard approach for differentiable functions is some form of gradient descent, resulting
in a sequence of updates
t+1=ttrf(t):
The performance of vanilla gradient descent, however, is hampered by the fact that it only makes use
of gradients and ignores second-order information. Classical optimization techniques correct this
behavior by rescaling the gradient step using curvature information, typically via the Hessian matrix
of second-order partial derivatives—although other choices such as the generalized Gauss-Newton
matrix or Fisher information matrix are possible.
Much of the modern work in optimization is based around designing update rules tailored to speciﬁc
classes of problems, with the types of problems of interest differing between different research


Much of the modern work in optimization is based around designing update rules tailored to speciﬁc
classes of problems, with the types of problems of interest differing between different research
communities. For example, in the deep learning community we have seen a proliferation of optimization methods specialized for high-dimensional, non-convex optimization problems. These include
momentum [Nesterov, 1983, Tseng, 1998], Rprop [Riedmiller and Braun, 1993], Adagrad [Duchi
et al., 2011], RMSprop [Tieleman and Hinton, 2012], and ADAM [Kingma and Ba, 2015]. More
focused methods can also be applied when more structure of the optimization problem is known
[Martens and Grosse, 2015]. In contrast, communities who focus on sparsity tend to favor very
different approaches [Donoho, 2006, Bach et al., 2012]. This is even more the case for combinatorial
optimization for which relaxations are often the norm [Nemhauser and Wolsey, 1988].
30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.arXiv:1606.04474v2  [cs.NE]  30 Nov 2016
optimizeroptimizeeparameter updates
error signalFigure 1: The optimizer (left) is provided with
performance of the optimizee (right) and proposes


optimizeroptimizeeparameter updates
error signalFigure 1: The optimizer (left) is provided with
performance of the optimizee (right) and proposes
updates to increase the optimizee’s performance.
[photos: Bobolas, 2009, Maley, 2011]This industry of optimizer design allows different communities to create optimization methods which exploit structure in their problems
of interest at the expense of potentially poor
performance on problems outside of that scope.
Moreover the No Free Lunch Theorems for Optimization [Wolpert and Macready, 1997] show
that in the setting of combinatorial optimization,
no algorithm is able to do better than a random
strategy in expectation. This suggests that specialization to a subclass of problems is in fact
theonly way that improved performance can be
achieved in general.
In this work we take a different tack and instead
propose to replace hand-designed update rules
with a learned update rule, which we call the optimizerg, speciﬁed by its own set of parameters
. This results in updates to the optimizee fof
the form
t+1=t+gt(rf(t);): (1)
A high level view of this process is shown in Figure 1. In what follows we will explicitly model


A high level view of this process is shown in Figure 1. In what follows we will explicitly model
the update rule gusing a recurrent neural network (RNN) which maintains its own state and hence
dynamically updates as a function of its iterates.
1.1 Transfer learning and generalization
The goal of this work is to develop a procedure for constructing a learning algorithm which performs
well on a particular class of optimization problems. Casting algorithm design as a learning problem
allows us to specify the class of problems we are interested in through example problem instances.
This is in contrast to the ordinary approach of characterizing properties of interesting problems
analytically and using these analytical insights to design learning algorithms by hand.
It is informative to consider the meaning of generalization in this framework. In ordinary statistical
learning we have a particular function of interest, whose behavior is constrained through a data set of
example function evaluations. In choosing a model we specify a set of inductive biases about how
we think the function of interest should behave at points we have not observed, and generalization
corresponds to the capacity to make predictions about the behavior of the target function at novel
points. In our setting the examples are themselves problem instances , which means generalization
corresponds to the ability to transfer knowledge between different problems. This reuse of problem
structure is commonly known as transfer learning , and is often treated as a subject in its own right.


corresponds to the ability to transfer knowledge between different problems. This reuse of problem
structure is commonly known as transfer learning , and is often treated as a subject in its own right.
However, by taking a meta-learning perspective, we can cast the problem of transfer learning as one
of generalization, which is much better studied in the machine learning community.
One of the great success stories of deep-learning is that we can rely on the ability of deep networks to
generalize to new examples by learning interesting sub-structures. In this work we aim to leverage
this generalization power, but also to lift it from simple supervised learning to the more general
setting of optimization.
1.2 A brief history and related work
The idea of using learning to learn ormeta-learning to acquire knowledge or inductive biases has a
long history [Thrun and Pratt, 1998]. More recently, Lake et al. [2016] have argued forcefully for
its importance as a building block in artiﬁcial intelligence. Similarly, Santoro et al. [2016] frame
multi-task learning as generalization, however unlike our approach they directly train a base learner
rather than a training algorithm. In general these ideas involve learning which occurs at two different
time scales: rapid learning within tasks and more gradual, meta learning across many different tasks.


rather than a training algorithm. In general these ideas involve learning which occurs at two different
time scales: rapid learning within tasks and more gradual, meta learning across many different tasks.
Perhaps the most general approach to meta-learning is that of Schmidhuber [1992, 1993]—building
on work from [Schmidhuber, 1987]—which considers networks that are able to modify their own
weights. Such a system is differentiable end-to-end, allowing both the network and the learning
2
algorithm to be trained jointly by gradient descent with few restrictions. However this generality
comes at the expense of making the learning rules very difﬁcult to train. Alternatively, the work
of Schmidhuber et al. [1997] uses the Success Story Algorithm to modify its search strategy rather
than gradient descent; a similar approach has been recently taken in Daniel et al. [2016] which uses
reinforcement learning to train a controller for selecting step-sizes.
Bengio et al. [1990, 1995] propose to learn updates which avoid back-propagation by using simple
parametric rules. In relation to the focus of this paper the work of Bengio et al. could be characterized
aslearning to learn without gradient descent by gradient descent . The work of Runarsson and


aslearning to learn without gradient descent by gradient descent . The work of Runarsson and
Jonsson [2000] builds upon this work by replacing the simple rule with a neural network.
Cotter and Conwell [1990], and later Younger et al. [1999], also show ﬁxed-weight recurrent neural
networks can exhibit dynamic behavior without need to modify their network weights. Similarly this
has been shown in a ﬁltering context [e.g. Feldkamp and Puskorius, 1998], which is directly related
to simple multi-timescale optimizers [Sutton, 1992, Schraudolph, 1999].
Finally, the work of Younger et al. [2001] and Hochreiter et al. [2001] connects these different threads
of research by allowing for the output of backpropagation from one network to feed into an additional
learning network, with both networks trained jointly. Our approach to meta-learning builds on this
work by modifying the network architecture of the optimizer in order to scale this approach to larger
neural-network optimization problems.
2 Learning to learn with recurrent neural networks
In this work we consider directly parameterizing the optimizer. As a result, in a slight abuse of notation
we will write the ﬁnal optimizee parameters (f;)as a function of the optimizer parameters and


we will write the ﬁnal optimizee parameters (f;)as a function of the optimizer parameters and
the function in question. We can then ask the question: What does it mean for an optimizer to be
good? Given a distribution of functions fwe will write the expected loss as
L() =Efh
f
(f;)i
: (2)
As noted earlier, we will take the update steps gtto be the output of a recurrent neural network m,
parameterized by , whose state we will denote explicitly with ht. Next, while the objective function
in(2)depends only on the ﬁnal parameter value, for training the optimizer it will be convenient to
have an objective that depends on the entire trajectory of optimization, for some horizon T,
L() =Ef"TX
t=1wtf(t)#
where t+1=t+gt;
gt
ht+1
=m(rt;ht;):(3)
Herewt2R0are arbitrary weights associated with each time-step and we will also use the notation
rt=rf(t). This formulation is equivalent to (2)whenwt=1[t=T], but later we will describe


rt=rf(t). This formulation is equivalent to (2)whenwt=1[t=T], but later we will describe
why using different weights can prove useful.
We can minimize the value of L()using gradient descent on . The gradient estimate @L()=@ can
be computed by sampling a random function fand applying backpropagation to the computational
graph in Figure 2. We allow gradients to ﬂow along the solid edges in the graph, but gradients
along the dashed edges are dropped. Ignoring gradients along the dashed edges amounts to making
the assumption that the gradients of the optimizee do not depend on the optimizer parameters, i.e.
@rt
@= 0. This assumption allows us to avoid computing second derivatives of f.
Examining the objective in (3)we see that the gradient is non-zero only for terms where wt6= 0. If
we usewt=1[t=T]to match the original problem, then gradients of trajectory preﬁxes are zero
and only the ﬁnal optimization step provides information for training the optimizer. This renders
Backpropagation Through Time (BPTT) inefﬁcient. We solve this problem by relaxing the objective


Backpropagation Through Time (BPTT) inefﬁcient. We solve this problem by relaxing the objective
such thatwt>0at intermediate points along the trajectory. This changes the objective function, but
allows us to train the optimizer on partial trajectories. For simplicity, in all our experiments we use
wt= 1for everyt.
3
OptimizeeOptimizert-2t-1t
mmm+++ft-1ftft-2∇t-2∇t-1∇tht-2ht-1htht+1gt-1gtθt-2θt-1θtθt+1gt-2Figure 2: Computational graph used for computing the gradient of the optimizer.
2.1 Coordinatewise LSTM optimizer
One challenge in applying RNNs in our setting is that we want to be able to optimize at least tens of
thousands of parameters. Optimizing at this scale with a fully connected RNN is not feasible as it
would require a huge hidden state and an enormous number of parameters. To avoid this difﬁculty we
will use an optimizer mwhich operates coordinatewise on the parameters of the objective function,
similar to other common update rules like RMSprop and ADAM. This coordinatewise network


will use an optimizer mwhich operates coordinatewise on the parameters of the objective function,
similar to other common update rules like RMSprop and ADAM. This coordinatewise network
architecture allows us to use a very small network that only looks at a single coordinate to deﬁne the
optimizer and share optimizer parameters across different parameters of the optimizee.
Different behavior on each coordinate is achieved by using separate activations for each objective
function parameter. In addition to allowing us to use a small network for this optimizer, this setup has
the nice effect of making the optimizer invariant to the order of parameters in the network, since the
same update rule is used independently on each coordinate.
LSTM1fLSTMn∇1θ1θn++
…∇n………
…
Figure 3: One step of an LSTM optimizer. All
LSTMs have shared parameters, but separate hidden states.We implement the update rule for each coordinate using a two-layer Long Short Term Memory
(LSTM) network [Hochreiter and Schmidhuber,
1997], using the now-standard forget gate architecture. The network takes as input the optimizee gradient for a single coordinate as well
as the previous hidden state and outputs the update for the corresponding optimizee parameter.
We will refer to this architecture, illustrated in


as the previous hidden state and outputs the update for the corresponding optimizee parameter.
We will refer to this architecture, illustrated in
Figure 3, as an LSTM optimizer.
The use of recurrence allows the LSTM to learn
dynamic update rules which integrate information from the history of gradients, similar to
momentum. This is known to have many desirable properties in convex optimization [see e.g.
Nesterov, 1983] and in fact many recent learning procedures—such as ADAM—use momentum in
their updates.
Preprocessing and postprocessing Optimizer inputs and outputs can have very different magnitudes depending on the class of function being optimized, but neural networks usually work robustly
only for inputs and outputs which are neither very small nor very large. In practice rescaling inputs
and outputs of an LSTM optimizer using suitable constants (shared across all timesteps and functions
f) is sufﬁcient to avoid this problem. In Appendix A we propose a different method of preprocessing
inputs to the optimizer inputs which is more robust and gives slightly better performance.
4
20 40 60 80 10010-1100101LossQuadratics
ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
Step100MNIST


ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
Step100MNIST
120 140 160 180 200MNIST, 200 stepsFigure 4: Comparisons between learned and hand-crafted optimizers performance. Learned optimizers are shown with solid lines and hand-crafted optimizers are shown with dashed lines. Units for the
yaxis in the MNIST plots are logits. Left: Performance of different optimizers on randomly sampled
10-dimensional quadratic functions. Center: the LSTM optimizer outperforms standard methods
training the base network on MNIST. Right: Learning curves for steps 100-200 by an optimizer
trained to optimize for 100 steps (continuation of center plot).
3 Experiments
In all experiments the trained optimizers use two-layer LSTMs with 20 hidden units in each layer.
Each optimizer is trained by minimizing Equation 3 using truncated BPTT as described in Section 2.
The minimization is performed using ADAM with a learning rate chosen by random search.
We use early stopping when training the optimizer in order to avoid overﬁtting the optimizer. After
each epoch (some ﬁxed number of learning steps) we freeze the optimizer parameters and evaluate its
performance. We pick the best optimizer (according to the ﬁnal validation loss) and report its average


performance. We pick the best optimizer (according to the ﬁnal validation loss) and report its average
performance on a number of freshly sampled test problems.
We compare our trained optimizers with standard optimizers used in Deep Learning: SGD, RMSprop,
ADAM, and Nesterov’s accelerated gradient (NAG). For each of these optimizer and each problem
we tuned the learning rate, and report results with the rate that gives the best ﬁnal error for each
problem. When an optimizer has more parameters than just a learning rate (e.g. decay coefﬁcients for
ADAM) we use the default values from the optim package in Torch7. Initial values of all optimizee
parameters were sampled from an IID Gaussian distribution.
3.1 Quadratic functions
In this experiment we consider training an optimizer on a simple class of synthetic 10-dimensional
quadratic functions. In particular we consider minimizing functions of the form
f() =kWyk2
2
for different 10x10 matrices Wand 10-dimensional vectors ywhose elements are drawn from an IID
Gaussian distribution. Optimizers were trained by optimizing random functions from this family and
tested on newly sampled functions from the same distribution. Each function was optimized for 100


Gaussian distribution. Optimizers were trained by optimizing random functions from this family and
tested on newly sampled functions from the same distribution. Each function was optimized for 100
steps and the trained optimizers were unrolled for 20 steps. We have not used any preprocessing, nor
postprocessing.
Learning curves for different optimizers, averaged over many functions, are shown in the left plot of
Figure 4. Each curve corresponds to the average performance of one optimization algorithm on many
test functions; the solid curve shows the learned optimizer performance and dashed curves show
the performance of the standard baseline optimizers. It is clear the learned optimizers substantially
outperform the baselines in this setting.
3.2 Training a small neural network on MNIST
In this experiment we test whether trainable optimizers can learn to optimize a small neural network
on MNIST, and also explore how the trained optimizers generalize to functions beyond those they
were trained on. To this end, we train the optimizer to optimize a base network and explore a series
of modiﬁcations to the network architecture and training procedure at test time.
5
20 40 60 80 100100LossMNIST, 40 units
ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
StepsMNIST, 2 layers


ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
StepsMNIST, 2 layers
20 40 60 80 100MNIST, ReLUFigure 5: Comparisons between learned and hand-crafted optimizers performance. Units for the
yaxis are logits. Left: Generalization to the different number of hidden units (40 instead of 20).
Center: Generalization to the different number of hidden layers (2 instead of 1). This optimization
problem is very hard, because the hidden layers are very narrow. Right: Training curves for an MLP
with 20 hidden units using ReLU activations. The LSTM optimizer was trained on an MLP with
sigmoid activations.
5 20 35 5010-1100Final Loss
LSTM
5 20 35 50
Hidden units
ADAM
5 20 35 50
NAG
Layers
1
2
Figure 6: Systematic study of ﬁnal MNIST performance as the optimizee architecture is varied,
using sigmoid non-linearities. The vertical dashed line in the left-most plot denotes the architecture
at which the LSTM is trained and the horizontal line shows the ﬁnal performance of the trained
optimizer in this setting.
In this setting the objective function f()is the cross entropy of a small MLP with parameters .


optimizer in this setting.
In this setting the objective function f()is the cross entropy of a small MLP with parameters .
The values of fas well as the gradients @f()=@are estimated using random minibatches of 128
examples. The base network is an MLP with one hidden layer of 20 units using a sigmoid activation
function. The only source of variability between different runs is the initial value 0and randomness
in minibatch selection. Each optimization was run for 100 steps and the trained optimizers were
unrolled for 20 steps. We used input preprocessing described in Appendix A and rescaled the outputs
of the LSTM by the factor 0:1.
Learning curves for the base network using different optimizers are displayed in the center plot of
Figure 4. In this experiment NAG, ADAM, and RMSprop exhibit roughly equivalent performance the
LSTM optimizer outperforms them by a signiﬁcant margin. The right plot in Figure 4 compares the
performance of the LSTM optimizer if it is allowed to run for 200 steps, despite having been trained
to optimize for 100 steps. In this comparison we re-used the LSTM optimizer from the previous
experiment, and here we see that the LSTM optimizer continues to outperform the baseline optimizers
on this task.


experiment, and here we see that the LSTM optimizer continues to outperform the baseline optimizers
on this task.
Generalization to different architectures Figure 5 shows three examples of applying the LSTM
optimizer to train networks with different architectures than the base network on which it was trained.
The modiﬁcations are (from left to right) (1) an MLP with 40 hidden units instead of 20, (2) a
network with two hidden layers instead of one, and (3) a network using ReLU activations instead of
sigmoid. In the ﬁrst two cases the LSTM optimizer generalizes well, and continues to outperform
the hand-designed baselines despite operating outside of its training regime. However, changing
the activation function to ReLU makes the dynamics of the learning procedure sufﬁciently different
that the learned optimizer is no longer able to generalize. Finally, in Figure 6 we show the results
of systematically varying the tested architecture; for the LSTM results we again used the optimizer
trained using 1 layer of 20 units and sigmoid non-linearities. Note that in this setting where the
6
200 400 600 800 1000100LossCIFAR-10
200 400 600 800 1000
Step100CIFAR-5
200 400 600 800 100010-1CIFAR-2


200 400 600 800 1000
Step100CIFAR-5
200 400 600 800 100010-1CIFAR-2
ADAM
RMSprop
SGD
NAG
LSTM
LSTM-subFigure 7: Optimization performance on the CIFAR-10 dataset and subsets. Shown on the left is the
LSTM optimizer versus various baselines trained on CIFAR-10 and tested on a held-out test set. The
two plots on the right are the performance of these optimizers on subsets of the CIFAR labels. The
additional optimizer LSTM-sub has been trained only on the heldout labels and is hence transferring
to a completely novel dataset.
test-set problems are similar enough to those in the training set we see even better generalization than
the baseline optimizers.
3.3 Training a convolutional network on CIFAR-10
Next we test the performance of the trained neural optimizers on optimizing classiﬁcation performance
for the CIFAR-10 dataset [Krizhevsky, 2009]. In these experiments we used a model with both
convolutional and feed-forward layers. In particular, the model used for these experiments includes
three convolutional layers with max pooling followed by a fully-connected layer with 32 hidden units;


three convolutional layers with max pooling followed by a fully-connected layer with 32 hidden units;
all non-linearities were ReLU activations with batch normalization.
The coordinatewise network decomposition introduced in Section 2.1—and used in the previous
experiment—utilizes a single LSTM architecture with shared weights, but separate hidden states,
for each optimizee parameter. We found that this decomposition was not sufﬁcient for the model
architecture introduced in this section due to the differences between the fully connected and convolutional layers. Instead we modify the optimizer by introducing two LSTMs: one proposes parameter
updates for the fully connected layers and the other updates the convolutional layer parameters. Like
the previous LSTM optimizer we still utilize a coordinatewise decomposition with shared weights
and individual hidden states, however LSTM weights are now shared only between parameters of the
same type (i.e. fully-connected vs. convolutional).
The performance of this trained optimizer compared against the baseline techniques is shown in
Figure 7. The left-most plot displays the results of using the optimizer to ﬁt a classiﬁer on a held-out
test set. The additional two plots on the right display the performance of the trained optimizer on


test set. The additional two plots on the right display the performance of the trained optimizer on
modiﬁed datasets which only contain a subset of the labels, i.e. the CIFAR-2 dataset only contains
data corresponding to 2 of the 10 labels. Additionally we include an optimizer LSTM-sub which was
only trained on the held-out labels.
In all these examples we can see that the LSTM optimizer learns much more quickly than the baseline
optimizers, with signiﬁcant boosts in performance for the CIFAR-5 and especially CIFAR-2 datsets.
We also see that the optimizers trained only on a disjoint subset of the data is hardly effected by this
difference and transfers well to the additional dataset.
3.4 Neural Art
The recent work on artistic style transfer using convolutional networks, or Neural Art [Gatys et al.,
2015], gives a natural testbed for our method, since each content and style image pair gives rise to a
different optimization problem. Each Neural Art problem starts from a content image ,c, and a style
image ,s, and is given by
f() =Lcontent (c;) +Lstyle(s;) +
Lreg()


image ,s, and is given by
f() =Lcontent (c;) +Lstyle(s;) +
Lreg()
The minimizer of fis the styled image . The ﬁrst two terms try to match the content and style of
the styled image to that of their ﬁrst argument, and the third term is a regularizer that encourages
smoothness in the styled image. Details can be found in [Gatys et al., 2015].
7
20 40 60 80 100 120
StepLossNeural art, training resolution
20 40 60 80 100 120
StepDouble resolution
ADAM
RMSprop
SGD
NAG
LSTMFigure 8: Optimization curves for Neural Art. Content images come from the test set, which was not
used during the LSTM optimizer training. Note: the y-axis is in log scale and we zoom in on the
interesting portion of this plot. Left: Applying the training style at the training resolution. Right:
Applying the test style at double the training resolution.
Figure 9: Examples of images styled using the LSTM optimizer. Each triple consists of the content
image (left), style (right) and image generated by the LSTM optimizer (center). Left: The result of


image (left), style (right) and image generated by the LSTM optimizer (center). Left: The result of
applying the training style at the training resolution to a test image. Right: The result of applying a
new style to a test image at double the resolution on which the optimizer was trained.
We train optimizers using only 1 style and 1800 content images taken from ImageNet [Deng et al.,
2009]. We randomly select 100 content images for testing and 20 content images for validation of
trained optimizers. We train the optimizer on 64x64 content images from ImageNet and one ﬁxed
style image. We then test how well it generalizes to a different style image and higher resolution
(128x128). Each image was optimized for 128 steps and trained optimizers were unrolled for 32
steps. Figure 9 shows the result of styling two different images using the LSTM optimizer. The
LSTM optimizer uses inputs preprocessing described in Appendix A and no postprocessing. See
Appendix C for additional images.
Figure 8 compares the performance of the LSTM optimizer to standard optimization algorithms. The
LSTM optimizer outperforms all standard optimizers if the resolution and style image are the same
as the ones on which it was trained. Moreover, it continues to perform very well when both the
resolution and style are changed at test time.


as the ones on which it was trained. Moreover, it continues to perform very well when both the
resolution and style are changed at test time.
Finally, in Appendix B we qualitatively examine the behavior of the step directions generated by the
learned optimizer.
4 Conclusion
We have shown how to cast the design of optimization algorithms as a learning problem, which
enables us to train optimizers that are specialized to particular classes of functions. Our experiments
have conﬁrmed that learned neural optimizers compare favorably against state-of-the-art optimization
methods used in deep learning. We witnessed a remarkable degree of transfer, with for example the
LSTM optimizer trained on 12,288 parameter neural art tasks being able to generalize to tasks with
49,152 parameters, different styles, and different content images all at the same time. We observed
similar impressive results when transferring to different architectures in the MNIST task.
The results on the CIFAR image labeling task show that the LSTM optimizers outperform handengineered optimizers when transferring to datasets drawn from the same data distribution.
References
F. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-inducing penalties. Foundations
and Trends in Machine Learning , 4(1):1–106, 2012.
8


and Trends in Machine Learning , 4(1):1–106, 2012.
8
S. Bengio, Y . Bengio, and J. Cloutier. On the search for new learning rules for ANNs. Neural Processing Letters ,
2(4):26–30, 1995.
Y . Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule . Université de Montréal, Département
d’informatique et de recherche opérationnelle, 1990.
Y . Bengio, N. Boulanger-Lewandowski, and R. Pascanu. Advances in optimizing recurrent networks. In
International Conference on Acoustics, Speech and Signal Processing , pages 8624–8628. IEEE, 2013.
F. Bobolas. brain-neurons, 2009. URL https://www.flickr.com/photos/fbobolas/3822222947 . Creative Commons Attribution-ShareAlike 2.0 Generic.
N. E. Cotter and P. R. Conwell. Fixed-weight networks can learn. In International Joint Conference on Neural
Networks , pages 553–559, 1990.
C. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In


Networks , pages 553–559, 1990.
C. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In
Association for the Advancement of Artiﬁcial Intelligence , 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database.
InComputer Vision and Pattern Recognition , pages 248–255. IEEE, 2009.
D. L. Donoho. Compressed sensing. Transactions on Information Theory , 52(4):1289–1306, 2006.
J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient methods for online learning and stochastic optimization.
Journal of Machine Learning Research , 12:2121–2159, 2011.
L. A. Feldkamp and G. V . Puskorius. A signal processing framework based on dynamic neural networks
with application to problems in adaptation, ﬁltering, and classiﬁcation. Proceedings of the IEEE , 86(11):
2259–2277, 1998.
L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv Report 1508.06576, 2015.


L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv Report 1508.06576, 2015.
A. Graves, G. Wayne, and I. Danihkela. Neural Turing machines. arXiv Report 1410.5401, 2014.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):1735–1780, 1997.
S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In International
Conference on Artiﬁcial Neural Networks , pages 87–94. Springer, 2001.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning
Representations , 2015.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like
people. arXiv Report 1604.00289, 2016.


people. arXiv Report 1604.00289, 2016.
T. Maley. neuron, 2011. URL https://www.flickr.com/photos/taylortotz101/6280077898 . Creative
Commons Attribution 2.0 Generic.
J. Martens and R. Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In
International Conference on Machine Learning , pages 2408–2417, 2015.
G. L. Nemhauser and L. A. Wolsey. Integer and combinatorial optimization . John Wiley & Sons, 1988.
Y . Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet
Mathematics Doklady , volume 27, pages 372–376, 1983.
J. Nocedal and S. Wright. Numerical optimization . Springer Science & Business Media, 2006.
M. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation learning: The RPROP
algorithm. In International Conference on Neural Networks , pages 586–591, 1993.
T. P. Runarsson and M. T. Jonsson. Evolution and design of distributed learning rules. In IEEE Symposium on


T. P. Runarsson and M. T. Jonsson. Evolution and design of distributed learning rules. In IEEE Symposium on
Combinations of Evolutionary Computation and Neural Networks , pages 59–63. IEEE, 2000.
A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented
neural networks. In International Conference on Machine Learning , 2016.
J. Schmidhuber. Evolutionary principles in self-referential learning; On learning how to learn: The meta-meta-...
hook. PhD thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
J. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks.
Neural Computation , 4(1):131–139, 1992.
J. Schmidhuber. A neural network that embeds its own meta-levels. In International Conference on Neural
Networks , pages 407–412. IEEE, 1993.
J. Schmidhuber, J. Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm, adaptive levin
search, and incremental self-improvement. Machine Learning , 28(1):105–130, 1997.


search, and incremental self-improvement. Machine Learning , 28(1):105–130, 1997.
N. N. Schraudolph. Local gain adaptation in stochastic gradient descent. In International Conference on
Artiﬁcial Neural Networks , volume 2, pages 569–574, 1999.
R. S. Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In Association for
the Advancement of Artiﬁcial Intelligence , pages 171–176, 1992.
S. Thrun and L. Pratt. Learning to learn . Springer Science & Business Media, 1998.
T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent
magnitude. COURSERA: Neural Networks for Machine Learning , 4:2, 2012.
P. Tseng. An incremental gradient (-projection) method with momentum term and adaptive stepsize rule. Journal
on Optimization , 8(2):506–531, 1998.
D. H. Wolpert and W. G. Macready. No free lunch theorems for optimization. Transactions on Evolutionary
Computation , 1(1):67–82, 1997.
9


Computation , 1(1):67–82, 1997.
9
A. S. Younger, P. R. Conwell, and N. E. Cotter. Fixed-weight on-line learning. Transactions on Neural Networks ,
10(2):272–283, 1999.
A. S. Younger, S. Hochreiter, and P. R. Conwell. Meta-learning with backpropagation. In International Joint
Conference on Neural Networks , 2001.
10
0 64 128−8−24
0 64 128−2616
LSTM
ADAM
SGDFigure 10: Updates proposed by different optimizers at different optimization steps for two different
coordinates.
A Gradient preprocessing
One potential challenge in training optimizers is that different input coordinates (i.e. the gradients
w.r.t. different optimizee parameters) can have very different magnitudes. This is indeed the case e.g.
when the optimizee is a neural network and different parameters correspond to weights in different
layers. This can make training an optimizer difﬁcult, because neural networks naturally disregard
small variations in input signals and concentrate on bigger input values.
To this aim we propose to preprocess the optimizer’s inputs. One solution would be to give the


small variations in input signals and concentrate on bigger input values.
To this aim we propose to preprocess the optimizer’s inputs. One solution would be to give the
optimizer (log(jrj);sgn(r))as an input, whereris the gradient in the current timestep. This has a
problem that log(jrj)diverges forr! 0. Therefore, we use the following preprocessing formula
rk!(
log(jrj)
p;sgn(r)
ifjrjep
(1;epr) otherwise
wherep>0is a parameter controlling how small gradients are disregarded (we use p= 10 in all our
experiments).
We noticed that just rescaling all inputs by an appropriate constant instead also works ﬁne, but the
proposed preprocessing seems to be more robust and gives slightly better results on some problems.
B Visualizations
Visualizing optimizers is inherently difﬁcult because their proposed updates are functions of the full
optimization trajectory. In this section we try to peek into the decisions made by the LSTM optimizer,
trained on the neural art task.
Histories of updates We select a single optimizee parameter (one color channel of one pixel in the


trained on the neural art task.
Histories of updates We select a single optimizee parameter (one color channel of one pixel in the
styled image) and trace the updates proposed to this coordinate by the LSTM optimizer over a single
trajectory of optimization. We also record the updates that would have been proposed by both SGD
and ADAM if they followed the same trajectory of iterates. Figure 10 shows the trajectory of updates
for two different optimizee parameters. From the plots it is clear that the trained optimizer makes
bigger updates than SGD and ADAM. It is also visible that it uses some kind of momentum, but its
updates are more noisy than those proposed by ADAM which may be interpreted as having a shorter
time-scale momentum.
Proposed update as a function of current gradient Another way to visualize the optimizer
behavior is to look at the proposed update gtfor a single coordinate as a function of the current
gradient evaluation rt. We follow the same procedure as in the previous experiment, and visualize
the proposed updates for a few selected time steps.
These results are shown in Figures 11–13. In these plots, the x-axis is the current value of the gradient
for the chosen coordinate, and the y-axis shows the update that each optimizer would propose should
the corresponding gradient value be observed. The history of gradient observations is the same for all


for the chosen coordinate, and the y-axis shows the update that each optimizer would propose should
the corresponding gradient value be observed. The history of gradient observations is the same for all
methods and follows the trajectory of the LSTM optimizer.
11
The shape of this function for the LSTM optimizer is often step-like, which is also the case for
ADAM. Surprisingly the step is sometimes in the opposite direction as for ADAM, i.e. the bigger the
gradient, the bigger the update.
C Neural Art
Shown below are additional examples of images styled using the LSTM optimizer. Each triple
consists of the content image (left), style (right) and image generated by the LSTM optimizer (center).
D Information sharing between coordinates
In previous sections we considered a coordinatewise architecture, which corresponds by analogy
to a learned version of RMSprop or ADAM. Although diagonal methods are quite effective in
practice, we can also consider learning more sophisticated optimizers that take the correlations
between coordinates into effect. To this end, we introduce a mechanism allowing different LSTMs to
communicate with each other.
Global averaging cells
The simplest solution is to designate a subset of the cells in each LSTM layer for communication.
These cells operate like normal LSTM cells, but their outgoing activations are averaged at each step


These cells operate like normal LSTM cells, but their outgoing activations are averaged at each step
across all coordinates. These global averaging cells (GACs) are sufﬁcient to allow the networks to
implement L2 gradient clipping [Bengio et al., 2013] assuming that each LSTM can compute the
square of the gradient. This architecture is denoted as an LSTM+GAC optimizer.
NTM-BFGS optimizer
We also consider augmenting the LSTM+GAC architecture with an external memory that is shared
between coordinates. Such a memory, if appropriately designed could allow the optimizer to learn
algorithms similar to (low-memory) approximations to Newton’s method, e.g. (L-)BFGS [see Nocedal
and Wright, 2006]. The reason for this interpretation is that such methods can be seen as a set of
independent processes working coordinatewise, but communicating through the inverse Hessian
approximation stored in the memory. We designed a memory architecture that, in theory, allows the
12
Step 1−10010
Step 2 Step 3 Step 4
Step 5−10010
Step 6 Step 7 Step 8
Step 9−10010
Step 10 Step 11 Step 12
Step 13−10010
Step 14 Step 15 Step 16
Step 17−10010


Step 6 Step 7 Step 8
Step 9−10010
Step 10 Step 11 Step 12
Step 13−10010
Step 14 Step 15 Step 16
Step 17−10010
Step 18 Step 19 Step 20
Step 21−10010
Step 22 Step 23 Step 24
Step 25−10010
Step 26 Step 27 Step 28
−400 0 400
Step 29−10010
−400 0 400
Step 30−400 0 400
Step 31−400 0 400
Step 32Figure 11: The proposed update direction for a single coordinate over 32 steps.
13
Step 1−10010
Step 2 Step 3 Step 4
Step 5−10010
Step 6 Step 7 Step 8
Step 9−10010
Step 10 Step 11 Step 12
Step 13−10010
Step 14 Step 15 Step 16
Step 17−10010
Step 18 Step 19 Step 20
Step 21−10010
Step 22 Step 23 Step 24
Step 25−10010
Step 26 Step 27 Step 28
−400 0 400
Step 29−10010
−400 0 400
Step 30−400 0 400
Step 31−400 0 400
Step 32Figure 12: The proposed update direction for a single coordinate over 32 steps.
14
Step 1−10010
Step 2 Step 3 Step 4
Step 5−10010
Step 6 Step 7 Step 8
Step 9−10010


14
Step 1−10010
Step 2 Step 3 Step 4
Step 5−10010
Step 6 Step 7 Step 8
Step 9−10010
Step 10 Step 11 Step 12
Step 13−10010
Step 14 Step 15 Step 16
Step 17−10010
Step 18 Step 19 Step 20
Step 21−10010
Step 22 Step 23 Step 24
Step 25−10010
Step 26 Step 27 Step 28
−400 0 400
Step 29−10010
−400 0 400
Step 30−400 0 400
Step 31−400 0 400
Step 32Figure 13: The proposed update direction for a single coordinate over 32 steps.
15
LSTMLSTMLSTMLSTMLSTMLSTMLSTMr1MLSTMr2r3…………rni1i2i3…………inx=
LSTMLSTMLSTMLSTMLSTMLSTMLSTMLSTMa1a2a3…………anb1b2b3…………bnOuterProductMtΔMtMt+1+=Figure 14: Left: NTM-BFGS read operation. Right: NTM-BFGS write operation.
network to simulate (L-)BFGS, motivated by the approximate Newton method BFGS, named for
Broyden, Fletcher, Goldfarb, and Shanno. We call this architecture an NTM-BFGS optimizer , because


Broyden, Fletcher, Goldfarb, and Shanno. We call this architecture an NTM-BFGS optimizer , because
its use of external memory is similar to the Neural Turing Machine [Graves et al., 2014]. The pivotal
differences between our construction and the NTM are (1) our memory allows only low-rank updates;
(2) the controller (including read/write heads) operates coordinatewise.
In BFGS an explicit estimate of the full (inverse) Hessian is built up from the sequence of observed
gradients. We can write a skeletonized version of the BFGS algorithm, using Mtto represent the
inverse Hessian approximation at iteration t, as follows
gt= read(Mt;t)
t+1=t+gt
Mt+1= write(Mt;t;gt):
Here we have packed up all of the details of the BFGS algorithm into the suggestively named read
andwrite operations, which operate on the inverse Hessian approximation Mt. In BFGS these
operations have speciﬁc forms, for example read(Mt;t) =Mtrf(t)is a speciﬁc matrix-vector
multiplication and the BFGS write operation corresponds to a particular low-rank update of Mt.


multiplication and the BFGS write operation corresponds to a particular low-rank update of Mt.
In this work we preserve the structure of the BFGS updates, but discard their particular form. More
speciﬁcally the read operation remains a matrix-vector multiplication but the form of the vector
used is learned. Similarly, the write operation remains a low-rank update, but the vectors involved
are also learned. Conveniently, this structure of interaction with a large dynamically updated state
corresponds in a fairly direct way to the architecture of a Neural Turing Machine (NTM), where Mt
corresponds to the NTM memory [Graves et al., 2014].
Our NTM-BFGS optimizer uses an LSTM+GAC as a controller; however, instead of producing the
update directly we attach one or more read and write heads to the controller. Each read head produces
a read vector rtwhich is combined with the memory to produce a read result itwhich is fed back
into the controller at the following time step. Each write head produces two outputs, a left write
vectoratand a right write vector bt. The two write vectors are used to update the memory state by
accumulating their outer product. The read and write operation for a single head is diagrammed in
Figure 14 and the way read and write heads are attached to the controller is depicted in Figure 15.


Figure 14 and the way read and write heads are attached to the controller is depicted in Figure 15.
In can be shown that NTM-BFGS with one read head and 3 write heads can simulate inverse Hessian
BFGS assuming that the controller can compute arbitrary (coordinatewise) functions and have access
to 2 GACs.
NTM-L-BFGS optimizer
In cases where memory is constrained we can follow the example of L-BFGS and maintain a low
rank approximation of the full memory ( vis. inverse Hessian). The simplest way to do this is to store a
sliding history of the left and right write vectors, allowing us to form the matrix vector multiplication
required by the read operation efﬁciently.
16
Controller         ReadWriteMtrtht+1Mt+1rt+1itatbt∇thtLSTM1LSTM2…gt
LSTMk∇kikakbkrkhkhkgkFigure 15: Left: Interaction between the controller and the external memory in NTM-BFGS. The
controller is composed of replicated coordinatewise LSTMs (possibly with GACs), but the read and
write operations are global across all coordinates. Right: A single LSTM for the kth coordinate in


write operations are global across all coordinates. Right: A single LSTM for the kth coordinate in
the NTM-BFGS controller. Note that here we have dropped the time index tto simplify notation.
17


Layer Normalization
Jimmy Lei Ba
University of Toronto
jimmy@psi.toronto.eduJamie Ryan Kiros
University of Toronto
rkiros@cs.toronto.eduGeoffrey E. Hinton
University of Toronto
and Google Inc.
hinton@cs.toronto.edu
Abstract
Training state-of-the-art, deep neural networks is computationally expensive. One
way to reduce the training time is to normalize the activities of the neurons. A
recently introduced technique called batch normalization uses the distribution of
the summed input to a neuron over a mini-batch of training cases to compute a
mean and variance which are then used to normalize the summed input to that
neuron on each training case. This signiﬁcantly reduces the training time in feedforward neural networks. However, the effect of batch normalization is dependent
on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by
computing the mean and variance used for normalization from all of the summed
inputs to the neurons in a layer on a single training case. Like batch normalization,
we also give each neuron its own adaptive bias and gain which are applied after
the normalization but before the non-linearity. Unlike batch normalization, layer

Layer Normalization

Summary:
Training state-of-the-art, deep neural networks is computationally expensive.
One way to reduce the training time is to normalize the activities of the
neurons. A recently introduced technique called batch normalization uses the
distribution of the summed input to a neuron over a mini-batch of training
cases to compute a mean and variance which are then used to normalize the
summed input to that neuron on each training case. This significantly reduces
the training time in feed-forward neural networks. However, the effect of batch
normalization is dependent on the mini-batch size and it is not obvious how to
apply it to recurrent neural networks. In this paper, we transpose batch
normalization into layer normalization by computing the mean and variance used
for normalization from all of the summed inputs to the neurons in a layer on a
single training case. Like batch normalization, we also give each neuron its
own adaptive bias and gain which are applied after the normalization but before
the non-linearity. Unlike batch normalization, layer normalization performs
exactly the same computation at training and test times. It is also
straightforward to apply to recurrent neural networks by computing the
normalization statistics separately at each time step. Layer normalization is
very effective at stabilizing the hidden state dynamics in recurrent networks.
Empirically, we show that layer normalization can substantially reduce the
training time compared with previously published techniques.

Jimmy Lei Ba
Jamie Ryan Kiros
Geoffrey E. Hinton

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Batch Normalized Recurrent


we also give each neuron its own adaptive bias and gain which are applied after
the normalization but before the non-linearity. Unlike batch normalization, layer
normalization performs exactly the same computation at training and test times.
It is also straightforward to apply to recurrent neural networks by computing the
normalization statistics separately at each time step. Layer normalization is very
effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time
compared with previously published techniques.
1 Introduction
Deep neural networks trained with some version of Stochastic Gradient Descent have been shown
to substantially outperform previous approaches on various supervised learning tasks in computer
vision [Krizhevsky et al., 2012] and speech processing [Hinton et al., 2012]. But state-of-the-art
deep neural networks often require many days of training. It is possible to speed-up the learning
by computing gradients for different subsets of the training cases on different machines or splitting
the neural network itself over many machines [Dean et al., 2012], but this can require a lot of communication and complex software. It also tends to lead to rapidly diminishing returns as the degree
of parallelization increases. An orthogonal approach is to modify the computations performed in
the forward pass of the neural net to make learning easier. Recently, batch normalization [Ioffe and

Layer Normalization

Batch Normalized Recurrent


the forward pass of the neural net to make learning easier. Recently, batch normalization [Ioffe and
Szegedy, 2015] has been proposed to reduce training time by including additional normalization
stages in deep neural networks. The normalization standardizes each summed input using its mean
and its standard deviation across the training data. Feedforward neural networks trained using batch
normalization converge faster even with simple SGD. In addition to training time improvement, the
stochasticity from the batch statistics serves as a regularizer during training.
Despite its simplicity, batch normalization requires running averages of the summed input statistics. In feed-forward networks with ﬁxed depth, it is straightforward to store the statistics separately
for each hidden layer. However, the summed inputs to the recurrent neurons in a recurrent neural network (RNN) often vary with the length of the sequence so applying batch normalization to
RNNs appears to require different statistics for different time-steps. Furthermore, batch normaliza-arXiv:1607.06450v1  [stat.ML]  21 Jul 2016
tion cannot be applied to online learning tasks or to extremely large distributed models where the
minibatches have to be small.
This paper introduces layer normalization, a simple normalization method to improve the training
speed for various neural network models. Unlike batch normalization, the proposed method directly

Layer Normalization

Batch Normalized Recurrent


This paper introduces layer normalization, a simple normalization method to improve the training
speed for various neural network models. Unlike batch normalization, the proposed method directly
estimates the normalization statistics from the summed inputs to the neurons within a hidden layer
so the normalization does not introduce any new dependencies between training cases. We show that
layer normalization works well for RNNs and improves both the training time and the generalization
performance of several existing RNN models.
2 Background
A feed-forward neural network is a non-linear mapping from a input pattern xto an output vector
y. Consider the lthhidden layer in a deep feed-forward, neural network, and let albe the vector
representation of the summed inputs to the neurons in that layer. The summed inputs are computed
through a linear projection with the weight matrix Wland the bottom-up inputs hlgiven as follows:
al
i=wl
i>hlhl+1
i=f(al
i+bl
i) (1)
wheref()is an element-wise non-linear function and wl
iis the incoming weights to the ithhidden
units andbl
iis the scalar bias parameter. The parameters in the neural network are learnt using
gradient-based optimization algorithms with the gradients being computed by back-propagation.

Layer Normalization

Batch Normalized Recurrent


units andbl
iis the scalar bias parameter. The parameters in the neural network are learnt using
gradient-based optimization algorithms with the gradients being computed by back-propagation.
One of the challenges of deep learning is that the gradients with respect to the weights in one layer
are highly dependent on the outputs of the neurons in the previous layer especially if these outputs
change in a highly correlated way. Batch normalization [Ioffe and Szegedy, 2015] was proposed
to reduce such undesirable “covariate shift”. The method normalizes the summed inputs to each
hidden unit over the training cases. Speciﬁcally, for the ithsummed input in the lthlayer, the batch
normalization method rescales the summed inputs according to their variances under the distribution
of the data
al
i=gl
i
l
i
al
il
i
l
i=E
xP(x)
al
i
l
i=r
E
xP(x)h
al
il
i2i
(2)
where al

Layer Normalization

Batch Normalized Recurrent


l
i=r
E
xP(x)h
al
il
i2i
(2)
where al
iis normalized summed inputs to the ithhidden unit in the lthlayer andgiis a gain parameter scaling the normalized activation before the non-linear activation function. Note the expectation
is under the whole training data distribution. It is typically impractical to compute the expectations
in Eq. (2) exactly, since it would require forward passes through the whole training dataset with the
current set of weights. Instead, andare estimated using the empirical samples from the current
mini-batch. This puts constraints on the size of a mini-batch and it is hard to apply to recurrent
neural networks.
3 Layer normalization
We now consider the layer normalization method which is designed to overcome the drawbacks of
batch normalization.
Notice that changes in the output of one layer will tend to cause highly correlated changes in the
summed inputs to the next layer, especially with ReLU units whose outputs can change by a lot.
This suggests the “covariate shift” problem can be reduced by ﬁxing the mean and the variance of
the summed inputs within each layer. We, thus, compute the layer normalization statistics over all
the hidden units in the same layer as follows:
l=1

Layer Normalization

Batch Normalized Recurrent


the summed inputs within each layer. We, thus, compute the layer normalization statistics over all
the hidden units in the same layer as follows:
l=1
HHX
i=1al
il=vuut1
HHX
i=1
al
il2(3)
whereHdenotes the number of hidden units in a layer. The difference between Eq. (2) and Eq. (3)
is that under layer normalization, all the hidden units in a layer share the same normalization terms
and, but different training cases have different normalization terms. Unlike batch normalization,
layer normaliztion does not impose any constraint on the size of a mini-batch and it can be used in
the pure online regime with batch size 1.
2
3.1 Layer normalized recurrent neural networks
The recent sequence to sequence models [Sutskever et al., 2014] utilize compact recurrent neural
networks to solve sequential prediction problems in natural language processing. It is common
among the NLP tasks to have different sentence lengths for different training cases. This is easy to
deal with in an RNN because the same weights are used at every time-step. But when we apply batch
normalization to an RNN in the obvious way, we need to to compute and store separate statistics for

Layer Normalization

Batch Normalized Recurrent


normalization to an RNN in the obvious way, we need to to compute and store separate statistics for
each time step in a sequence. This is problematic if a test sequence is longer than any of the training
sequences. Layer normalization does not have such problem because its normalization terms depend
only on the summed inputs to a layer at the current time-step. It also has only one set of gain and
bias parameters shared over all time-steps.
In a standard RNN, the summed inputs in the recurrent layer are computed from the current input
xtand previous vector of hidden states ht1which are computed as at=Whhht1+Wxhxt. The
layer normalized recurrent layer re-centers and re-scales its activations using the extra normalization
terms similar to Eq. (3):
ht=fhg
t
att
+bi
t=1
HHX
i=1at
it=vuut1
HHX
i=1(at
it)2(4)
whereWhhis the recurrent hidden to hidden weights and Wxhare the bottom up input to hidden
weights.is the element-wise multiplication between two vectors. bandgare deﬁned as the bias
and gain parameters of the same dimension as ht.

Layer Normalization

Batch Normalized Recurrent


weights.is the element-wise multiplication between two vectors. bandgare deﬁned as the bias
and gain parameters of the same dimension as ht.
In a standard RNN, there is a tendency for the average magnitude of the summed inputs to the recurrent units to either grow or shrink at every time-step, leading to exploding or vanishing gradients. In
a layer normalized RNN, the normalization terms make it invariant to re-scaling all of the summed
inputs to a layer, which results in much more stable hidden-to-hidden dynamics.
4 Related work
Batch normalization has been previously extended to recurrent neural networks [Laurent et al., 2015,
Amodei et al., 2015, Cooijmans et al., 2016]. The previous work [Cooijmans et al., 2016] suggests
the best performance of recurrent batch normalization is obtained by keeping independent normalization statistics for each time-step. The authors show that initializing the gain parameter in the
recurrent batch normalization layer to 0.1 makes signiﬁcant difference in the ﬁnal performance of
the model. Our work is also related to weight normalization [Salimans and Kingma, 2016]. In
weight normalization, instead of the variance, the L2 norm of the incoming weights is used to

Layer Normalization

Batch Normalized Recurrent


weight normalization, instead of the variance, the L2 norm of the incoming weights is used to
normalize the summed inputs to a neuron. Applying either weight normalization or batch normalization using expected statistics is equivalent to have a different parameterization of the original
feed-forward neural network. Re-parameterization in the ReLU network was studied in the Pathnormalized SGD [Neyshabur et al., 2015]. Our proposed layer normalization method, however, is
not a re-parameterization of the original neural network. The layer normalized model, thus, has
different invariance properties than the other methods, that we will study in the following section.
5 Analysis
In this section, we investigate the invariance properties of different normalization schemes.
5.1 Invariance under weights and data transformations
The proposed layer normalization is related to batch normalization and weight normalization. Although, their normalization scalars are computed differently, these methods can be summarized as
normalizing the summed inputs aito a neuron through the two scalars and. They also learn an
adaptive bias band gaingfor each neuron after the normalization.
hi=f(gi
i(aii) +bi) (5)
Note that for layer normalization and batch normalization, andis computed according to Eq. 2

Layer Normalization

Batch Normalized Recurrent


i(aii) +bi) (5)
Note that for layer normalization and batch normalization, andis computed according to Eq. 2
and 3. In weight normalization, is 0, and=kwk2.
3
Weight matrix Weight matrix Weight vector Dataset Dataset Single training case
re-scaling re-centering re-scaling re-scaling re-centering re-scaling
Batch norm Invariant No Invariant Invariant Invariant No
Weight norm Invariant No Invariant No No No
Layer norm Invariant Invariant No Invariant No Invariant
Table 1: Invariance properties under the normalization methods.
Table 1 highlights the following invariance results for three normalization methods.
Weight re-scaling and re-centering: First, observe that under batch normalization and weight
normalization, any re-scaling to the incoming weights wiof a single neuron has no effect on the
normalized summed inputs to a neuron. To be precise, under batch and weight normalization, if
the weight vector is scaled by , the two scalar andwill also be scaled by . The normalized
summed inputs stays the same before and after scaling. So the batch and weight normalization are

Layer Normalization

Batch Normalized Recurrent


summed inputs stays the same before and after scaling. So the batch and weight normalization are
invariant to the re-scaling of the weights. Layer normalization, on the other hand, is not invariant
to the individual scaling of the single weight vectors. Instead, layer normalization is invariant to
scaling of the entire weight matrix and invariant to a shift to all of the incoming weights in the
weight matrix. Let there be two sets of model parameters ,0whose weight matrices WandW0
differ by a scaling factor and all of the incoming weights in W0are also shifted by a constant
vector
, that isW0=W+1
>. Under layer normalization, the two models effectively compute
the same output:
h0=f(g
0(W0x0) +b) =f(g
0
(W+1
>)x0
+b)
=f(g
(Wx) +b) =h: (6)
Notice that if normalization is only applied to the input before the weights, the model will not be
invariant to re-scaling and re-centering of the weights.
Data re-scaling and re-centering: We can show that all the normalization methods are invariant

Layer Normalization

Batch Normalized Recurrent


Data re-scaling and re-centering: We can show that all the normalization methods are invariant
to re-scaling the dataset by verifying that the summed inputs of neurons stays constant under the
changes. Furthermore, layer normalization is invariant to re-scaling of individual training cases,
because the normalization scalars andin Eq. (3) only depend on the current input data. Let x0
be a new data point obtained by re-scaling xby. Then we have,
h0
i=f(gi
0
w>
ix00
+bi) =f(gi

w>
ix
+bi) =hi: (7)
It is easy to see re-scaling individual data points does not change the model’s prediction under layer
normalization. Similar to the re-centering of the weight matrix in layer normalization, we can also
show that batch normalization is invariant to re-centering of the dataset.
5.2 Geometry of parameter space during learning
We have investigated the invariance of the model’s prediction under re-centering and re-scaling of
the parameters. Learning, however, can behave very differently under different parameterizations,

Layer Normalization

Batch Normalized Recurrent


the parameters. Learning, however, can behave very differently under different parameterizations,
even though the models express the same underlying function. In this section, we analyze learning
behavior through the geometry and the manifold of the parameter space. We show that the normalization scalar can implicitly reduce learning rate and makes learning more stable.
5.2.1 Riemannian metric
The learnable parameters in a statistical model form a smooth manifold that consists of all possible
input-output relations of the model. For models whose output is a probability distribution, a natural
way to measure the separation of two points on this manifold is the Kullback-Leibler divergence
between their model output distributions. Under the KL divergence metric, the parameter space is a
Riemannian manifold.
The curvature of a Riemannian manifold is entirely captured by its Riemannian metric, whose
quadratic form is denoted as ds2. That is the inﬁnitesimal distance in the tangent space at a point
in the parameter space. Intuitively, it measures the changes in the model output from the parameter
space along a tangent direction. The Riemannian metric under KL was previously studied [Amari,
1998] and was shown to be well approximated under second order Taylor expansion using the Fisher
4
information matrix:
ds2= D KL

Layer Normalization

Batch Normalized Recurrent


1998] and was shown to be well approximated under second order Taylor expansion using the Fisher
4
information matrix:
ds2= D KL
P(yjx;)kP(yjx;+)
1
2>F(); (8)
F() = E
xP(x);yP(yjx)"
@logP(yjx;)
@@logP(yjx;)
@>#
; (9)
where,is a small change to the parameters. The Riemannian metric above presents a geometric
view of parameter spaces. The following analysis of the Riemannian metric provides some insight
into how normalization methods could help in training neural networks.
5.2.2 The geometry of normalized generalized linear models
We focus our geometric analysis on the generalized linear model. The results from the following
analysis can be easily applied to understand deep neural networks with block-diagonal approximation to the Fisher information matrix, where each block corresponds to the parameters for a single
neuron.
A generalized linear model (GLM) can be regarded as parameterizing an output distribution from
the exponential family using a weight vector wand bias scalar b. To be consistent with the previous

Layer Normalization

Batch Normalized Recurrent


A generalized linear model (GLM) can be regarded as parameterizing an output distribution from
the exponential family using a weight vector wand bias scalar b. To be consistent with the previous
sections, the log likelihood of the GLM can be written using the summed inputs aas the following:
logP(yjx;w;b) =(a+b)y(a+b)
+c(y;); (10)
E[yjx] =f(a+b) =f(w>x+b);Var[yjx] =f0(a+b); (11)
where,f()is the transfer function that is the analog of the non-linearity in neural networks, f0()
is the derivative of the transfer function, ()is a real valued function and c()is the log partition function. is a constant that scales the output variance. Assume a H-dimensional output
vector y= [y1;y2;;yH]is modeled using Hindependent GLMs and logP(yjx;W;b) =PH
i=1logP(yijx;wi;bi). LetWbe the weight matrix whose rows are the weight vectors of the

Layer Normalization

Batch Normalized Recurrent


i=1logP(yijx;wi;bi). LetWbe the weight matrix whose rows are the weight vectors of the
individual GLMs, bdenote the bias vector of length Handvec()denote the Kronecker vector operator. The Fisher information matrix for the multi-dimensional GLM with respect to its parameters
= [w>
1;b1;;w>
H;bH]>= vec([W;b]>)is simply the expected Kronecker product of the data
features and the output covariance matrix:
F() = E
xP(x)Cov[yjx]
2

xx>x
x>1
: (12)
We obtain normalized GLMs by applying the normalization methods to the summed inputs ain
the original model through and. Without loss of generality, we denote Fas the Fisher information matrix under the normalized multi-dimensional GLM with the additional gain parameters
= vec([W;b;g]>):
F() =2
664F11 F1H
.........
FH1 FHH3
775;Fij=E
xP(x)2
664Cov[yi; yjjx]

Layer Normalization

Batch Normalized Recurrent


FH1 FHH3
775;Fij=E
xP(x)2
664Cov[yi; yjjx]
22
664gigj
iji>
jigi
iigi(ajj)
ij
>
jgj
j1ajj
j
>
jgj(aii)
ijaii
i(aii)(ajj)
ij3
7753
775
(13)
i=x@i
@wiaii
i@i
@wi: (14)
Implicit learning rate reduction through the growth of the weight vector: Notice that, comparing to standard GLM, the block Fijalong the weight vector widirection is scaled by the gain
parameters and the normalization scalar i. If the norm of the weight vector wigrows twice as large,
even though the model’s output remains the same, the Fisher information matrix will be different.
The curvature along the widirection will change by a factor of1
2because the iwill also be twice

Layer Normalization

Batch Normalized Recurrent


The curvature along the widirection will change by a factor of1
2because the iwill also be twice
as large. As a result, for the same parameter update in the normalized model, the norm of the weight
vector effectively controls the learning rate for the weight vector. During learning, it is harder to
change the orientation of the weight vector with large norm. The normalization methods, therefore,
5
050100150200250300
iteration x 30034353637383940414243mean Recall@1Image Retrieval (Validation)
Order-Embedding + LN
Order-Embedding(a) Recall@1
050100150200250300
iteration x 3007172737475767778mean Recall@5Image Retrieval (Validation)
Order-Embedding + LN
Order-Embedding (b) Recall@5
050100150200250300
iteration x 30084858687888990mean Recall@10Image Retrieval (Validation)
Order-Embedding + LN
Order-Embedding (c) Recall@10
Figure 1: Recall@K curves using order-embeddings with and without layer normalization.
MSCOCO
Caption Retrieval Image Retrieval

Layer Normalization

Batch Normalized Recurrent


Figure 1: Recall@K curves using order-embeddings with and without layer normalization.
MSCOCO
Caption Retrieval Image Retrieval
Model R@1 R@5 R@10 Mean rR@1 R@5 R@10 Mean r
Sym [Vendrov et al., 2016] 45.4 88.7 5.8 36.3 85.8 9.0
OE [Vendrov et al., 2016] 46.7 88.9 5.7 37.9 85.9 8.1
OE (ours) 46.6 79.3 89.1 5.2 37.8 73.6 85.7 7.9
OE + LN 48.5 80.6 89.8 5.1 38.9 74.3 86.3 7.6
Table 2: Average results across 5 test splits for caption and image retrieval. R@K is Recall@K
(high is good). Mean ris the mean rank (low is good). Sym corresponds to the symmetric baseline
while OE indicates order-embeddings.
have an implicit “early stopping” effect on the weight vectors and help to stabilize learning towards
convergence.
Learning the magnitude of incoming weights: In normalized models, the magnitude of the incoming weights is explicitly parameterized by the gain parameters. We compare how the model output

Layer Normalization

Batch Normalized Recurrent


convergence.
Learning the magnitude of incoming weights: In normalized models, the magnitude of the incoming weights is explicitly parameterized by the gain parameters. We compare how the model output
changes between updating the gain parameters in the normalized GLM and updating the magnitude
of the equivalent weights under original parameterization during learning. The direction along the
gain parameters in Fcaptures the geometry for the magnitude of the incoming weights. We show
that Riemannian metric along the magnitude of the incoming weights for the standard GLM is scaled
by the norm of its input, whereas learning the gain parameters for the batch normalized and layer
normalized models depends only on the magnitude of the prediction error. Learning the magnitude
of incoming weights in the normalized model is therefore, more robust to the scaling of the input
and its parameters than in the standard model. See Appendix for detailed derivations.
6 Experimental results
We perform experiments with layer normalization on 6 tasks, with a focus on recurrent neural networks: image-sentence ranking, question-answering, contextual language modelling, generative
modelling, handwriting sequence generation and MNIST classiﬁcation. Unless otherwise noted,
the default initialization of layer normalization is to set the adaptive gains to 1and the biases to 0in
the experiments.
6.1 Order embeddings of images and language

Layer Normalization

Batch Normalized Recurrent


the default initialization of layer normalization is to set the adaptive gains to 1and the biases to 0in
the experiments.
6.1 Order embeddings of images and language
In this experiment, we apply layer normalization to the recently proposed order-embeddings model
of Vendrov et al. [2016] for learning a joint embedding space of images and sentences. We follow
the same experimental protocol as Vendrov et al. [2016] and modify their publicly available code
to incorporate layer normalization1which utilizes Theano [Team et al., 2016]. Images and sentences from the Microsoft COCO dataset [Lin et al., 2014] are embedded into a common vector
space, where a GRU [Cho et al., 2014] is used to encode sentences and the outputs of a pre-trained
VGG ConvNet [Simonyan and Zisserman, 2015] (10-crop) are used to encode images. The orderembedding model represents images and sentences as a 2-level partial ordering and replaces the
cosine similarity scoring function used in Kiros et al. [2014] with an asymmetric one.
1https://github.com/ivendrov/order-embedding
6
0 100 200 300 400 500 600 700 800
training steps (thousands)0.40.50.60.70.80.91.0validation error rateAttentive reader

Layer Normalization

Batch Normalized Recurrent


0 100 200 300 400 500 600 700 800
training steps (thousands)0.40.50.60.70.80.91.0validation error rateAttentive reader
LSTM
BN-LSTM
BN-everywhere
LN-LSTMFigure 2: Validation curves for the attentive reader model. BN results are taken from [Cooijmans
et al., 2016].
We trained two models: the baseline order-embedding model as well as the same model with layer
normalization applied to the GRU. After every 300 iterations, we compute Recall@K (R@K) values
on a held out validation set and save the model whenever R@K improves. The best performing
models are then evaluated on 5 separate test sets, each containing 1000 images and 5000 captions,
for which the mean results are reported. Both models use Adam [Kingma and Ba, 2014] with the
same initial hyperparameters and both models are trained using the same architectural choices as
used in Vendrov et al. [2016]. We refer the reader to the appendix for a description of how layer
normalization is applied to GRU.
Figure 1illustrates the validation curves of the models, with and without layer normalization. We
plot R@1, R@5 and R@10 for the image retrieval task. We observe that layer normalization offers

Layer Normalization

Batch Normalized Recurrent


plot R@1, R@5 and R@10 for the image retrieval task. We observe that layer normalization offers
a per-iteration speedup across all metrics and converges to its best validation model in 60% of the
time it takes the baseline model to do so. In Table 2, the test set results are reported from which we
observe that layer normalization also results in improved generalization over the original model. The
results we report are state-of-the-art for RNN embedding models, with only the structure-preserving
model of Wang et al. [2016] reporting better results on this task. However, they evaluate under
different conditions (1 test set instead of the mean over 5) and are thus not directly comparable.
6.2 Teaching machines to read and comprehend
In order to compare layer normalization to the recently proposed recurrent batch normalization
[Cooijmans et al., 2016], we train an unidirectional attentive reader model on the CNN corpus both
introduced by Hermann et al. [2015]. This is a question-answering task where a query description
about a passage must be answered by ﬁlling in a blank. The data is anonymized such that entities

Layer Normalization

Batch Normalized Recurrent


about a passage must be answered by ﬁlling in a blank. The data is anonymized such that entities
are given randomized tokens to prevent degenerate solutions, which are consistently permuted during training and evaluation. We follow the same experimental protocol as Cooijmans et al. [2016]
and modify their public code to incorporate layer normalization2which uses Theano [Team et al.,
2016]. We obtained the pre-processed dataset used by Cooijmans et al. [2016] which differs from
the original experiments of Hermann et al. [2015] in that each passage is limited to 4 sentences.
In Cooijmans et al. [2016], two variants of recurrent batch normalization are used: one where BN
is only applied to the LSTM while the other applies BN everywhere throughout the model. In our
experiment, we only apply layer normalization within the LSTM.
The results of this experiment are shown in Figure 2. We observe that layer normalization not only
trains faster but converges to a better validation result over both the baseline and BN variants. In
Cooijmans et al. [2016], it is argued that the scale parameter in BN must be carefully chosen and is
set to 0.1 in their experiments. We experimented with layer normalization for both 1.0 and 0.1 scale

Layer Normalization

Batch Normalized Recurrent


set to 0.1 in their experiments. We experimented with layer normalization for both 1.0 and 0.1 scale
initialization and found that the former model performed signiﬁcantly better. This demonstrates that
layer normalization is not sensitive to the initial scale in the same way that recurrent BN is.3
6.3 Skip-thought vectors
Skip-thoughts [Kiros et al., 2015] is a generalization of the skip-gram model [Mikolov et al., 2013]
for learning unsupervised distributed sentence representations. Given contiguous text, a sentence is
2https://github.com/cooijmanstim/Attentive_reader/tree/bn
3We only produce results on the validation set, as in the case of Cooijmans et al. [2016]
7
5 10 15 20
iteration x 5000082.082.583.083.584.084.585.085.586.0Pearson x 100
Skip-Thoughts + LN
Skip-Thoughts
Original(a) SICK(r)
5 10 15 20
iteration x 500002728293031323334MSE x 100
Skip-Thoughts + LN
Skip-Thoughts
Original (b) SICK(MSE)
5 10 15 20

Layer Normalization

Batch Normalized Recurrent


Skip-Thoughts + LN
Skip-Thoughts
Original (b) SICK(MSE)
5 10 15 20
iteration x 5000070727476788082Accuracy
Skip-Thoughts + LN
Skip-Thoughts
Original (c) MR
5 10 15 20
iteration x 5000074767880828486Accuracy
Skip-Thoughts + LN
Skip-Thoughts
Original
(d) CR
5 10 15 20
iteration x 5000090.090.591.091.592.092.593.093.594.094.5Accuracy
Skip-Thoughts + LN
Skip-Thoughts
Original (e) SUBJ
5 10 15 20
iteration x 50000838485868788899091Accuracy
Skip-Thoughts + LN
Skip-Thoughts
Original (f) MPQA
Figure 3: Performance of skip-thought vectors with and without layer normalization on downstream
tasks as a function of training iterations. The original lines are the reported results in [Kiros et al.,
2015]. Plots with error use 10-fold cross validation. Best seen in color.
Method SICK( r) SICK() SICK( MSE ) MR CR SUBJ MPQA

Layer Normalization

Batch Normalized Recurrent


Method SICK( r) SICK() SICK( MSE ) MR CR SUBJ MPQA
Original [Kiros et al., 2015] 0.848 0.778 0.287 75.5 79.3 92.1 86.9
Ours 0.842 0.767 0.298 77.3 81.8 92.6 87.9
Ours + LN 0.854 0.785 0.277 79.5 82.6 93.4 89.0
Ours + LN y 0.858 0.788 0.270 79.4 83.1 93.7 89.3
Table 3: Skip-thoughts results. The ﬁrst two evaluation columns indicate Pearson and Spearman correlation, the third is mean squared error and the remaining indicate classiﬁcation accuracy. Higher is
better for all evaluations except MSE. Our models were trained for 1M iterations with the exception
of (y) which was trained for 1 month (approximately 1.7M iterations)
encoded with a encoder RNN and decoder RNNs are used to predict the surrounding sentences.
Kiros et al. [2015] showed that this model could produce generic sentence representations that

Layer Normalization

Batch Normalized Recurrent


Kiros et al. [2015] showed that this model could produce generic sentence representations that
perform well on several tasks without being ﬁne-tuned. However, training this model is timeconsuming, requiring several days of training in order to produce meaningful results.
In this experiment we determine to what effect layer normalization can speed up training. Using the
publicly available code of Kiros et al. [2015]4, we train two models on the BookCorpus dataset [Zhu
et al., 2015]: one with and one without layer normalization. These experiments are performed with
Theano [Team et al., 2016]. We adhere to the experimental setup used in Kiros et al. [2015], training
a 2400-dimensional sentence encoder with the same hyperparameters. Given the size of the states
used, it is conceivable layer normalization would produce slower per-iteration updates than without.
However, we found that provided CNMeM5is used, there was no signiﬁcant difference between the
two models. We checkpoint both models after every 50,000 iterations and evaluate their performance
on ﬁve tasks: semantic-relatedness (SICK) [Marelli et al., 2014], movie review sentiment (MR)
[Pang and Lee, 2005], customer product reviews (CR) [Hu and Liu, 2004], subjectivity/objectivity

Layer Normalization

Batch Normalized Recurrent


[Pang and Lee, 2005], customer product reviews (CR) [Hu and Liu, 2004], subjectivity/objectivity
classiﬁcation (SUBJ) [Pang and Lee, 2004] and opinion polarity (MPQA) [Wiebe et al., 2005]. We
plot the performance of both models for each checkpoint on all tasks to determine whether the
performance rate can be improved with LN.
The experimental results are illustrated in Figure 3. We observe that applying layer normalization
results both in speedup over the baseline as well as better ﬁnal results after 1M iterations are performed as shown in Table 3. We also let the model with layer normalization train for a total of a
month, resulting in further performance gains across all but one task. We note that the performance
4https://github.com/ryankiros/skip-thoughts
5https://github.com/NVIDIA/cnmem
8
100101102103
Updates x 200−900−800−700−600−500−400−300−200−1000Negative Log LikelihoodBaseline test
Baseline train
LN test
LN trainFigure 5: Handwriting sequence generation model negative log likelihood with and without layer
normalization. The models are trained with mini-batch size of 8 and sequence length of 500,

Layer Normalization

Batch Normalized Recurrent


LN trainFigure 5: Handwriting sequence generation model negative log likelihood with and without layer
normalization. The models are trained with mini-batch size of 8 and sequence length of 500,
differences between the original reported results and ours are likely due to the fact that the publicly
available code does not condition at each timestep of the decoder, where the original model does.
6.4 Modeling binarized MNIST using DRAW
0 20 40 60 80 100
Epoch80859095100Test Variational BoundBaseline
WN
LN
Figure 4: DRAW model test negative log likelihood with and without
layer normalization.We also experimented with the generative modeling on the
MNIST dataset. Deep Recurrent Attention Writer (DRAW)
[Gregor et al., 2015] has previously achieved the state-of-theart performance on modeling the distribution of MNIST digits. The model uses a differential attention mechanism and
a recurrent neural network to sequentially generate pieces of
an image. We evaluate the effect of layer normalization on
a DRAW model using 64 glimpses and 256 LSTM hidden
units. The model is trained with the default setting of Adam
[Kingma and Ba, 2014] optimizer and the minibatch size of
128. Previous publications on binarized MNIST have used
various training protocols to generate their datasets. In this

Layer Normalization

Batch Normalized Recurrent


128. Previous publications on binarized MNIST have used
various training protocols to generate their datasets. In this
experiment, we used the ﬁxed binarization from Larochelle
and Murray [2011]. The dataset has been split into 50,000
training, 10,000 validation and 10,000 test images.
Figure 4 shows the test variational bound for the ﬁrst 100
epoch. It highlights the speedup beneﬁt of applying layer normalization that the layer normalized DRAW converges almost twice as fast than the baseline model.
After 200 epoches, the baseline model converges to a variational log likelihood of 82.36 nats on the
test data and the layer normalization model obtains 82.09 nats.
6.5 Handwriting sequence generation
The previous experiments mostly examine RNNs on NLP tasks whose lengths are in the range of 10
to 40. To show the effectiveness of layer normalization on longer sequences, we performed handwriting generation tasks using the IAM Online Handwriting Database [Liwicki and Bunke, 2005].
IAM-OnDB consists of handwritten lines collected from 221 different writers. When given the input
character string, the goal is to predict a sequence of x and y pen co-ordinates of the corresponding

Layer Normalization

Batch Normalized Recurrent


character string, the goal is to predict a sequence of x and y pen co-ordinates of the corresponding
handwriting line on the whiteboard. There are, in total, 12179 handwriting line sequences. The input
string is typically more than 25 characters and the average handwriting line has a length around 700.
We used the same model architecture as in Section (5.2) of Graves [2013]. The model architecture
consists of three hidden layers of 400 LSTM cells, which produce 20 bivariate Gaussian mixture
components at the output layer, and a size 3 input layer. The character sequence was encoded with
one-hot vectors, and hence the window vectors were size 57. A mixture of 10 Gaussian functions
was used for the window parameters, requiring a size 30 parameter vector. The total number of
weights was increased to approximately 3.7M. The model is trained using mini-batches of size 8
and the Adam [Kingma and Ba, 2014] optimizer.
The combination of small mini-batch size and very long sequences makes it important to have very
stable hidden dynamics. Figure 5 shows that layer normalization converges to a comparable log
likelihood as the baseline model but is much faster.
9
0 10 20 30 40 50 60
Epoch10-710-610-510-410-310-210-1100Train NLL
BatchNorm bz128

Layer Normalization

Batch Normalized Recurrent


9
0 10 20 30 40 50 60
Epoch10-710-610-510-410-310-210-1100Train NLL
BatchNorm bz128
Baseline bz128
LayerNorm bz128
0 10 20 30 40 50 60
Epoch0.0050.0100.0150.0200.025Test Err.
BatchNorm bz128
Baseline bz128
LayerNorm bz128
0 10 20 30 40 50 60
Epoch10-710-610-510-410-310-210-1100Train NLL
LayerNorm bz4
Baseline bz4
BatchNorm bz4
0 10 20 30 40 50 60
Epoch0.0050.0100.0150.0200.025Test Err.
LayerNorm bz4
Baseline bz4
BatchNorm bz4Figure 6: Permutation invariant MNIST 784-1000-1000-10 model negative log likelihood and test
error with layer normalization and batch normalization. (Left) The models are trained with batchsize of 128. (Right) The models are trained with batch-size of 4.
6.6 Permutation invariant MNIST
In addition to RNNs, we investigated layer normalization in feed-forward networks. We show how

Layer Normalization

Batch Normalized Recurrent


6.6 Permutation invariant MNIST
In addition to RNNs, we investigated layer normalization in feed-forward networks. We show how
layer normalization compares with batch normalization on the well-studied permutation invariant
MNIST classiﬁcation problem. From the previous analysis, layer normalization is invariant to input
re-scaling which is desirable for the internal hidden layers. But this is unnecessary for the logit
outputs where the prediction conﬁdence is determined by the scale of the logits. We only apply
layer normalization to the fully-connected hidden layers that excludes the last softmax layer.
All the models were trained using 55000 training data points and the Adam [Kingma and Ba, 2014]
optimizer. For the smaller batch-size, the variance term for batch normalization is computed using
the unbiased estimator. The experimental results from Figure 6 highlight that layer normalization is
robust to the batch-sizes and exhibits a faster training convergence comparing to batch normalization
that is applied to all layers.
6.7 Convolutional Networks
We have also experimented with convolutional neural networks. In our preliminary experiments, we
observed that layer normalization offers a speedup over the baseline model without normalization,
but batch normalization outperforms the other methods. With fully connected layers, all the hidden

Layer Normalization

Batch Normalized Recurrent


observed that layer normalization offers a speedup over the baseline model without normalization,
but batch normalization outperforms the other methods. With fully connected layers, all the hidden
units in a layer tend to make similar contributions to the ﬁnal prediction and re-centering and rescaling the summed inputs to a layer works well. However, the assumption of similar contributions
is no longer true for convolutional neural networks. The large number of the hidden units whose
receptive ﬁelds lie near the boundary of the image are rarely turned on and thus have very different
statistics from the rest of the hidden units within the same layer. We think further research is needed
to make layer normalization work well in ConvNets.
7 Conclusion
In this paper, we introduced layer normalization to speed-up the training of neural networks. We
provided a theoretical analysis that compared the invariance properties of layer normalization with
batch normalization and weight normalization. We showed that layer normalization is invariant to
per training-case feature shifting and scaling.
Empirically, we showed that recurrent neural networks beneﬁt the most from the proposed method
especially for long sequences and small mini-batches.
Acknowledgments
This research was funded by grants from NSERC, CFI, and Google.
10
References

Layer Normalization

Batch Normalized Recurrent


especially for long sequences and small mini-batches.
Acknowledgments
This research was funded by grants from NSERC, CFI, and Google.
10
References
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classiﬁcation with deep convolutional neural
networks. In NIPS , 2012.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior,
Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in
speech recognition: The shared views of four research groups. IEEE , 2012.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker,
Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In NIPS , 2012.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing
internal covariate shift. ICML , 2015.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In
Advances in neural information processing systems , pages 3104–3112, 2014.

Layer Normalization

Batch Normalized Recurrent


Advances in neural information processing systems , pages 3104–3112, 2014.
C´esar Laurent, Gabriel Pereyra, Phil ´emon Brakel, Ying Zhang, and Yoshua Bengio. Batch normalized recurrent
neural networks. arXiv preprint arXiv:1510.01378 , 2015.
Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen,
Mike Chrzanowski, Adam Coates, Greg Diamos, et al. Deep speech 2: End-to-end speech recognition in
english and mandarin. arXiv preprint arXiv:1512.02595 , 2015.
Tim Cooijmans, Nicolas Ballas, C ´esar Laurent, and Aaron Courville. Recurrent batch normalization. arXiv
preprint arXiv:1603.09025 , 2016.
Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868 , 2016.
Behnam Neyshabur, Ruslan R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep

Layer Normalization

Batch Normalized Recurrent


Behnam Neyshabur, Ruslan R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep
neural networks. In Advances in Neural Information Processing Systems , pages 2413–2421, 2015.
Shun-Ichi Amari. Natural gradient works efﬁciently in learning. Neural computation , 1998.
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language.
ICLR , 2016.
The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller,
Dzmitry Bahdanau, Nicolas Ballas, Fr ´ed´eric Bastien, Justin Bayer, Anatoly Belikov, et al. Theano: A python
framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 , 2016.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and
C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV , 2014.

Layer Normalization

Batch Normalized Recurrent