S. Araki, and T. Nakatani. The ntt chime-3 system: Advances in speech enhancement and recognition for
mobile multi-microphone devices. In IEEE ASRU , 2015.
[70] W. Zaremba and I. Sutskever. Learning to execute. abs/1410.4615, 2014. http://arxiv.org/abs/1410.4615.
A Scalability improvements
In this section, we discuss some of our scalability improvements in more detail.
A.1 Node and cluster architecture
The software stack runs on a compute dense node built from 2Intel CPUs and 8NVIDIA Titan
X GPUs, with peak single-precision computational throughput of 53teraFLOP/second. Each node
also has 384GB of CPU memory and an 8TB storage volume built from two 4 TB hard disks in
RAID-0 conﬁguration. We use the CPU memory to cache our input data so that we are not directly
exposed to the low bandwidth and high latency of spinning disks. We replicate our English and
Mandarin datasets on each node’s local hard disk. This allows us to use our network only for weight
updates and avoids having to rely on centralized ﬁle servers.
GPUGPUGPUGPUGPUGPUGPUGPUPLXPLXPLXCPUCPUPLX


updates and avoids having to rely on centralized ﬁle servers.
GPUGPUGPUGPUGPUGPUGPUGPUPLXPLXPLXCPUCPUPLX
Figure 8: Schematic of our training node where PLX indicates a PCI switch and the dotted box includes all
devices that are connected by the same PCI root complex.
Figure 8 shows a schematic diagram of one our nodes, where all devices connected by the same PCI
root complex are encapsulated in a dotted box. We have tried to maximize the number of GPUs
within the root complex for faster communication between GPUs using GPUDirect. This allows us
to use an efﬁcient communication mechanism to transfer gradient matrices between GPUs.
26
All the nodes in our cluster are connected through Fourteen Data Rate (FDR) Inﬁniband which is
primarily used for gradient transfer during back-propagation.
A.2 GPU Implementation of CTC Loss Function
The CTC loss function that we use to train our models has two passes: forward and backward, and
the gradient computation involves element-wise addition of two matrices, and, generated during
the forward and backward passes respectively. Finally, we sum the gradients using the character in

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Summary:
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech--two vastly different languages.
Because it replaces entire pipelines of hand-engineered components with neural
networks, end-to-end learning allows us to handle a diverse variety of speech
including noisy environments, accents and different languages. Key to our
approach is our application of HPC techniques, resulting in a 7x speedup over
our previous system. Because of this efficiency, experiments that previously
took weeks now run in days. This enables us to iterate more quickly to identify
superior architectures and algorithms. As a result, in several cases, our
system is competitive with the transcription of human workers when benchmarked
on standard datasets. Finally, using a technique called Batch Dispatch with
GPUs in the data center, we show that our system can be inexpensively deployed
in an online setting, delivering low latency when serving users at scale.

Dario Amodei
Rishita Anubhai
Eric Battenberg
Carl Case
Jared Casper
Bryan Catanzaro
Jingdong Chen
Mike Chrzanowski
Adam Coates
Greg Diamos
Erich Elsen
Jesse Engel
Linxi Fan
Christopher Fougner
Tony Han
Awni Hannun
Billy Jun
Patrick LeGresley
Libby Lin
Sharan Narang
Andrew Ng
Sherjil Ozair
Ryan Prenger
Jonathan Raiman
Sanjeev Satheesh
David Seetapun
Shubho Sengupta
Yi Wang
Zhiqian Wang
Chong Wang
Bo Xiao
Dani Yogatama
Jun Zhan
Zhenyao Zhu


the gradient computation involves element-wise addition of two matrices, and, generated during
the forward and backward passes respectively. Finally, we sum the gradients using the character in
the utterance label as the key, to generate one gradient per character. These gradients are then backpropagated through the network. The input to the CTC loss function are probabilities calculated by
the softmax function which can be very small, so we compute in logprobability space for better
numerical stability.
The forward pass of the CTC algorithm calculates the matrix, which has Srows and Tcolumns,
where S= 2(L+ 1) . The variable Lis the number of characters in the label and Tis the number
of time-steps in the utterance. Our CPU-based implementation of the CTC algorithm assigns one
thread to each utterance label in a minibatch, performing the CTC calculation for the utterances in
parallel. Each thread calculates the relevant entries of the matrix sequentially. This is inefﬁcient for
two reasons.
Firstly, since the remainder of our network is computed on the GPU, the output of the softmax
function has to be copied to the CPU for CTC calculation. The gradient matrices from the CTC
function then has to be copied back to the GPU for backpropagation. For languages like Mandarin


function then has to be copied back to the GPU for backpropagation. For languages like Mandarin
with large character sets, these matrices have hundreds of millions of entries, making this copy
expensive. Furthermore, we need as much interconnect bandwidth as possible for synchronizing the
gradient updates with data parallelism, so this copy incurs a substantial opportunity cost.
Secondly, although entries in each column of the matrix can be computed in parallel, the number
of entries to calculate in each column depends both on the column and the number of repeated
characters in the utterance label. Due to this complexity, the CPU implementation does not use
SIMD parallelism optimally, making the computation inefﬁcient.
We wrote a GPU-based implementation of CTC in order to overcome these two problems. The key
insight behind our implementation is that we can compute all elements in each column of the
matrix, rather than just the valid entries. If we do so, Figure 9 shows that invalid elements either
contain a ﬁnite garbage value ( G), or1 (I), when we use a special summation function that adds
probabilities in logspace that discards inputs that are 1. This summation is shown in Figure 9
where arrows incident on a circle are inputs and the result is stored in the circle. However, when we


where arrows incident on a circle are inputs and the result is stored in the circle. However, when we
compute the ﬁnal gradient by element-wise summing and, all ﬁnite garbage values will be added
with a corresponding 1 value from the other matrix, which results in 1, effectively ignoring the
garbage value and computing the correct result. One important observation is that this element-wise
sum ofandis a simple sum and does not use our summation function.
To compute the gradient, we take each column of the matrix generated from element-wise addition
ofandmatrices, and do a key-value reduction using the character as key, using the ModernGPU
library [5]. This means elements of the column corresponding to the same character will sum up
their values. In the example shown in Figure 9, the blank character, B, is the only repeated character
and at some columns, say for t= 1 oft= 2, both valid elements (gray) and 1 correspond to
it. Since our summation function in logspace effectively ignores the 1 elements, only the valid
elements are combined in the reduction.
In our GPU implementation, we map each utterance in the minibatch to a CUDA thread block .


elements are combined in the reduction.
In our GPU implementation, we map each utterance in the minibatch to a CUDA thread block .
Since there are no dependencies between the elements of a column, all of them can be computed in
parallel by the threads in a threadblock. There are dependencies between columns, since the column
corresponding to time-step t+ 1cannot be computed before the column corresponding to time-step
t. The reverse happens when computing the matrix, when column corresponding to time-step
tcannot be computed before the column corresponding to time-step t+ 1. Thus, in both cases,
columns are processed sequentially by the thread block.
Mapping the forward and backward passes to corresponding CUDA kernels is straightforward since
there are no data dependencies between elements of a column. The kernel that does the backward
pass also computes the gradient. However, since the gradients must be summed up based on the label
27
IIIIATCIIIIIGGGGGGGG
G1234T-3T-2T-1T
GGGGATCGGGGGIIIIIIII
I↵BBBBBBBBFigure 9: Forward and backward pass for GPU implementation of CTC. Gray circles contain valid values,


GGGGATCGGGGGIIIIIIII
I↵BBBBBBBBFigure 9: Forward and backward pass for GPU implementation of CTC. Gray circles contain valid values,
circle with Icontain1 and circle with Gcontain garbage values that are ﬁnite. Bstand for the blank character
that the CTC algorithm adds to the input utterance label. Column labels on top show different time-steps going
from1toT.
values, with each character as key, we must deal with data dependencies due to repeated characters
in an utterance label. For languages with small character sets like English, this happens with high
probability. Even if there are no repeated characters, the CTC algorithm adds L+ 1blank characters
to the utterance label. We solve this problem by performing a key-value sort, where the keys are the
characters in the utterance label, and the values are the indices of each character in the utterance.
After sorting, all occurrences of a given character are arranged in contiguous segments. We only
need to do the sort once for each utterance. The indices generated by the sort are then used to
sequentially sum up the gradients for each character. This sum is done once per column and in
parallel over all characters in the utterance. Amortizing the cost of key-value sort over Tcolumns is


parallel over all characters in the utterance. Amortizing the cost of key-value sort over Tcolumns is
a key insight that makes the gradient calculation fast.
Our GPU implementation uses fast shared memory and registers to achieve high performance when
performing this task. Both forward and backward kernels store the matrix in shared memory .
Since shared memory is a limited resource, it is not possible to store the entire matrix. However,
as we go backward in time, we only need to keep one column of the matrix as we compute the
gradient, adding element-wise the column of the matrix with the corresponding column of the
matrix. Due to on-chip memory space constraints, we read the output of the softmax function
directly from off-chip global memory .
Due to inaccuracies in ﬂoating-point arithmetic, especially in transcendental functions, our GPU and
CPU implementation are not bit-wise identical. This is not an impediment in practice, since both
implementations train models equally well when coupled with the technique of sorting utterances
by length mentioned in Section 3.3.
28


Published as a conference paper at ICLR 2017
RECURRENT BATCH NORMALIZATION
Tim Cooijmans, Nicolas Ballas, César Laurent, Ça ˘glar Gülçehre & Aaron Courville
MILA - Université de Montréal
firstname.lastname@umontreal.ca
ABSTRACT
We propose a reparameterization of LSTM that brings the beneﬁts of batch normalization to recurrent neural networks. Whereas previous works only apply batch
normalization to the input-to-hidden transformation of RNNs, we demonstrate that
it is both possible and beneﬁcial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps.
We evaluate our proposal on various sequential problems such as sequence classiﬁcation, language modeling and question answering. Our empirical results show
that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.
1 I NTRODUCTION
Recurrent neural network architectures such as LSTM (Hochreiter & Schmidhuber, 1997) and
GRU (Cho et al., 2014) have recently exhibited state-of-the-art performance on a wide range of

Recurrent Batch Normalization

Summary:
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.

Tim Cooijmans
Nicolas Ballas
César Laurent
Çağlar Gülçehre
Aaron Courville

Unitary Evolution Recurrent Neural Networks

Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton

Neural Machine Translation by Jointly Learning to Align and Translate

Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Hierarchical Multiscale Recurrent Neural Networks

Junyoung Chung, Sungjin Ahn, Yoshua Bengio


GRU (Cho et al., 2014) have recently exhibited state-of-the-art performance on a wide range of
complex sequential problems including speech recognition Amodei et al. (2015), machine translation (Bahdanau et al., 2015) and image and video captioning (Xu et al., 2015; Yao et al., 2015).
Top-performing models, however, are based on very high-capacity networks that are computationally intensive and costly to train. Effective optimization of recurrent neural networks is thus an
active area of study (Pascanu et al., 2012; Martens & Sutskever, 2011; Ollivier, 2013).
It is well-known that for deep feed-forward neural networks, covariate shift (Shimodaira, 2000; Ioffe
& Szegedy, 2015) degrades the efﬁciency of training. Covariate shift is a change in the distribution
of the inputs to a model. This occurs continuously during training of feed-forward neural networks,
where changing the parameters of a layer affects the distribution of the inputs to all layers above it.
As a result, the upper layers are continually adapting to the shifting input distribution and unable
to learn effectively. This internal covariate shift (Ioffe & Szegedy, 2015) may play an especially
important role in recurrent neural networks, which resemble very deep feed-forward networks.

Recurrent Batch Normalization

Layer Normalization


important role in recurrent neural networks, which resemble very deep feed-forward networks.
Batch normalization (Ioffe & Szegedy, 2015) is a recently proposed technique for controlling the
distributions of feed-forward neural network activations, thereby reducing internal covariate shift. It
involves standardizing the activations going into each layer, enforcing their means and variances to
be invariant to changes in the parameters of the underlying layers. This effectively decouples each
layer’s parameters from those of other layers, leading to a better-conditioned optimization problem.
Indeed, deep neural networks trained with batch normalization converge signiﬁcantly faster and
generalize better.
Although batch normalization has demonstrated signiﬁcant training speed-ups and generalization
beneﬁts in feed-forward networks, it is proven to be difﬁcult to apply in recurrent architectures (Laurent et al., 2016; Amodei et al., 2015). It has found limited use in stacked RNNs, where the normalization is applied “vertically”, i.e. to the input of each RNN, but not “horizontally” between
timesteps. RNNs are deeper in the time direction, and as such batch normalization would be most

Recurrent Batch Normalization

Layer Normalization


timesteps. RNNs are deeper in the time direction, and as such batch normalization would be most
beneﬁcial when applied horizontally. However, Laurent et al. (2016) hypothesized that applying
batch normalization in this way hurts training because of exploding gradients due to repeated rescaling.
Our ﬁndings run counter to this hypothesis. We show that it is both possible and highly beneﬁcial to
apply batch normalization in the hidden-to-hidden transition of recurrent models. In particular, we
describe a reparameterization of LSTM (Section 3) that involves batch normalization and demonstrate that it is easier to optimize and generalizes better. In addition, we empirically analyze the
1arXiv:1603.09025v5  [cs.LG]  28 Feb 2017
Published as a conference paper at ICLR 2017
gradient backpropagation and show that proper initialization of the batch normalization parameters
is crucial to avoiding vanishing gradient (Section 4). We evaluate our proposal on several sequential problems and show (Section 5) that our LSTM reparameterization consistently outperforms the
LSTM baseline across tasks, in terms of both time to convergence and performance.
Liao & Poggio (2016) simultaneously investigated batch normalization in recurrent neural networks,

Recurrent Batch Normalization

Layer Normalization


LSTM baseline across tasks, in terms of both time to convergence and performance.
Liao & Poggio (2016) simultaneously investigated batch normalization in recurrent neural networks,
albeit only for very short sequences (10 steps). Ba et al. (2016) independently developed a variant
of batch normalization that is also applicable to recurrent neural networks and delivers similar improvements as our method.
2 P REREQUISITES
2.1 LSTM
Long Short-Term Memory (LSTM) networks are an instance of a more general class of recurrent
neural networks (RNNs), which we review brieﬂy in this paper. Given an input sequence X=
(x1;x2;:::;xT), an RNN deﬁnes a sequence of hidden states htaccording to
ht=(Whht1+Wxxt+b); (1)
where Wh2Rdhdh;Wx2Rdxdh;b2Rdhand the initial state h02Rdhare model parameters. A popular choice for the activation function ()istanh .
RNNs are popular in sequence modeling thanks to their natural ability to process variable-length

Recurrent Batch Normalization

Layer Normalization


RNNs are popular in sequence modeling thanks to their natural ability to process variable-length
sequences. However, training RNNs using ﬁrst-order stochastic gradient descent (SGD) is notoriously difﬁcult due to the well-known problem of exploding/vanishing gradients (Bengio et al., 1994;
Hochreiter, 1991; Pascanu et al., 2012). Gradient vanishing occurs when states htare not inﬂuenced
by small changes in much earlier states h,t, preventing learning of long-term dependencies in
the input data. Although learning long-term dependencies is fundamentally difﬁcult (Bengio et al.,
1994), its effects can be mitigated through architectural variations such as LSTM (Hochreiter &
Schmidhuber, 1997), GRU (Cho et al., 2014) and iRNN/uRNN (Le et al., 2015; Arjovsky et al.,
2015).
In what follows, we focus on the LSTM architecture (Hochreiter & Schmidhuber, 1997) with recurrent transition given by
0
BB@~ft
~it
~ot
~gt1
CCA=Whht1+Wxxt+b (2)

Recurrent Batch Normalization

Layer Normalization


0
BB@~ft
~it
~ot
~gt1
CCA=Whht1+Wxxt+b (2)
ct=(~ft)ct1+(~it)tanh( ~gt) (3)
ht=(~ot)tanh(ct); (4)
where Wh2Rdh4dh;WxRdx4dh;b2R4dhand the initial states h02Rdh;c02Rdhare
model parameters. is the logistic sigmoid function, and the operator denotes the Hadamard
product.
The LSTM differs from simple RNNs in that it has an additional memory cellctwhose update is
nearly linear which allows the gradient to ﬂow back through time more easily. In addition, unlike
the RNN which overwrites its content at each timestep, the update of the LSTM cell is regulated by
a set of gates. The forget gate ftdetermines the extent to which information is carried over from the
previous timestep, and the input gate itcontrols the ﬂow of information from the current input xt.
The output gate otallows the model to read from the cell. This carefully controlled interaction with

Recurrent Batch Normalization

Layer Normalization


The output gate otallows the model to read from the cell. This carefully controlled interaction with
the cell is what allows the LSTM to robustly retain information for long periods of time.
2.2 B ATCH NORMALIZATION
Covariate shift (Shimodaira, 2000) is a phenomenon in machine learning where the features presented to a model change in distribution. In order for learning to succeed in the presence of covariate shift, the model’s parameters must be adjusted not just to learn the concept at hand but also to
adapt to the changing distribution of the inputs. In deep neural networks, this problem manifests as
2
Published as a conference paper at ICLR 2017
internal covariate shift (Ioffe & Szegedy, 2015), where changing the parameters of a layer affects
the distribution of the inputs to all layers above it.
Batch Normalization (Ioffe & Szegedy, 2015) is a recently proposed network reparameterization
which aims to reduce internal covariate shift. It does so by standardizing the activations using
empirical estimates of their means and standard deviations. However, it does not decorrelate the
activations due to the computationally costly matrix inversion. The batch normalizing transform is
as follows:
BN(h;
;) =+
hbE[h]q

Recurrent Batch Normalization

Layer Normalization


as follows:
BN(h;
;) =+
hbE[h]q
dVar[h] +(5)
where h2Rdis the vector of (pre)activations to be normalized,
2Rd;2Rdare model
parameters that determine the mean and standard deviation of the normalized activation, and 2R
is a regularization hyperparameter. The division should be understood to proceed elementwise.
At training time, the statistics E[h]andVar[h]are estimated by the sample mean and sample variance of the current minibatch. This allows for backpropagation through the statistics, preserving the
convergence properties of stochastic gradient descent. During inference, the statistics are typically
estimated based on the entire training set, so as to produce a deterministic prediction.
3 B ATCH -NORMALIZED LSTM
This section introduces a reparameterization of LSTM that takes advantage of batch normalization.
Contrary to Laurent et al. (2016); Amodei et al. (2015), we leverage batch normalization in both
the input-to-hidden andthe hidden-to-hidden transformations. We introduce the batch-normalizing
transform BN(;
;)into the LSTM as follows:
0
BB@~ft

Recurrent Batch Normalization

Layer Normalization


transform BN(;
;)into the LSTM as follows:
0
BB@~ft
~it
~ot
~gt1
CCA= BN( Whht1;
h;h) + BN( Wxxt;
x;x) +b (6)
ct=(~ft)ct1+(~it)tanh( ~gt) (7)
ht=(~ot)tanh(BN( ct;
c;c)) (8)
In our formulation, we normalize the recurrent term Whht1and the input term Wxxtseparately.
Normalizing these terms individually gives the model better control over the relative contribution
of the terms using the
hand
xparameters. We set h=x=0to avoid unnecessary redundancy, instead relying on the pre-existing parameter vector bto account for both biases. In order to
leave the LSTM dynamics intact and preserve the gradient ﬂow through ct, we do not apply batch
normalization in the cell update.
The batch normalization transform relies on batch statistics to standardize the LSTM activations. It
would seem natural to share the statistics that are used for normalization across time, just as recurrent

Recurrent Batch Normalization

Layer Normalization


The batch normalization transform relies on batch statistics to standardize the LSTM activations. It
would seem natural to share the statistics that are used for normalization across time, just as recurrent
neural networks share their parameters over time. However, we ﬁnd that simply averaging statistics
over time severely degrades performance. Although LSTM activations do converge to a stationary
distribution, we observe that their statistics during the initial transient differ signiﬁcantly (see Figure 5 in Appendix A). Consequently, we recommend using separate statistics for each timestep to
preserve information of the initial transient phase in the activations.1
Generalizing the model to sequences longer than those seen during training is straightforward thanks
to the rapid convergence of the activations to their steady-state distributions (cf. Figure 5). For our
experiments we estimate the population statistics separately for each timestep 1;:::;Tmax where
1Note that we separate only the statistics over time and not the
andparameters.
3
Published as a conference paper at ICLR 2017
Tmax is the length of the longest training sequence. When at test time we need to generalize beyond
Tmax, we use the population statistic of time Tmax for all time steps beyond it.
During training we estimate the statistics across the minibatch, independently for each timestep. At

Recurrent Batch Normalization

Layer Normalization


Tmax, we use the population statistic of time Tmax for all time steps beyond it.
During training we estimate the statistics across the minibatch, independently for each timestep. At
test time we use estimates obtained by averaging the minibatch estimates over the training set.
4 I NITIALIZING
FOR GRADIENT FLOW
Although batch normalization allows for easy control of the pre-activation variance through the
parameters, common practice is to normalize to unit variance. We suspect that the previous difﬁculties with recurrent batch normalization reported in Laurent et al. (2016); Amodei et al. (2015) are
largely due to improper initialization of the batch normalization parameters, and
in particular. In
this section we demonstrate the impact of
on gradient ﬂow.
0 100 200 300 400 500 600 700 800
t10-2610-2410-2210-2010-1810-1610-1410-1210-1010-810-610-410-2100||dloss/dh_t||_2RNN gradient propagation
gamma=0.10
gamma=0.20
gamma=0.30
gamma=0.40
gamma=0.50
gamma=0.60
gamma=0.70
gamma=0.80

Recurrent Batch Normalization

Layer Normalization


gamma=0.40
gamma=0.50
gamma=0.60
gamma=0.70
gamma=0.80
gamma=0.90
gamma=1.00
(a) We visualize the gradient ﬂow through a batchnormalized tanh RNN as a function of
. High
variance causes vanishing gradient.
0.0 0.2 0.4 0.6 0.8 1.0
input standard deviation0.00.20.40.60.81.0expected derivative (and IQR range)derivative through tanh(b) We show the empirical expected derivative and
interquartile range of tanh nonlinearity as a function of input variance. High variance causes saturation, which decreases the expected derivative.
Figure 1: Inﬂuence of pre-activation variance on gradient propagation.
In Figure 1(a), we show how the pre-activation variance impacts gradient propagation in a simple
RNN on the sequential MNIST task described in Section 5.1. Since backpropagation operates in
reverse, the plot is best read from right to left. The quantity plotted is the norm of the gradient
of the loss with respect to the hidden state at different time steps. For large values of
, the norm

Recurrent Batch Normalization

Layer Normalization


of the loss with respect to the hidden state at different time steps. For large values of
, the norm
quickly goes to zero as gradient is propagated back in time. For small values of
the norm is nearly
constant.
To demonstrate what we think is the cause of this vanishing, we drew samples xfrom a set of
centered Gaussian distributions with standard deviation ranging from 0 to 1, and computed the
derivative tanh0(x) = 1tanh2(x)2[0;1]for each. Figure 1(b) shows the empirical distribution
of the derivative as a function of standard deviation. When the input standard deviation is low, the
input tends to be close to the origin where the derivative is close to 1. As the standard deviation
increases, the expected derivative decreases as the input is more likely to be in the saturation regime.
At unit standard deviation, the expected derivative is much smaller than 1.
We conjecture that this is what causes the gradient to vanish, and recommend initializing
to a
small value. In our trials we found that values of 0.01 or lower caused instabilities during training.
Our choice of 0.1 seems to work well across different tasks.
5 E XPERIMENTS

Recurrent Batch Normalization

Layer Normalization


Our choice of 0.1 seems to work well across different tasks.
5 E XPERIMENTS
This section presents an empirical evaluation of the proposed batch-normalized LSTM on four different tasks. Note that for all the experiments, we initialize the batch normalization scale and shift
parameters
andto0:1and0respectively.
4
Published as a conference paper at ICLR 2017
0 20000 40000 60000 80000 100000
Training Iteration0.00.20.40.60.81.0AccuracyPixel-by-Pixel MNIST (Validation Set)
lstm
bn_lstm
0 20000 40000 60000 80000 100000
Training Iteration0.10.20.30.40.50.60.70.80.91.0AccuracyPixel-by-Pixel Permuted-MNIST (Validation Set)
lstm
bn_lstm
Figure 2: Accuracy on the validation set for the pixel by pixel MNIST classiﬁcation tasks. The
batch-normalized LSTM is able to converge faster relatively to a baseline LSTM. Batch-normalized
LSTM also shows some improve generalization on the permuted sequential MNIST that require to
preserve long-term memory information.
5.1 S EQUENTIAL MNIST

Recurrent Batch Normalization

Layer Normalization


LSTM also shows some improve generalization on the permuted sequential MNIST that require to
preserve long-term memory information.
5.1 S EQUENTIAL MNIST
We evaluate our batch-normalized LSTM on a sequential version of the MNIST classiﬁcation
task (Le et al., 2015). The model processes each image one pixel at a time and ﬁnally predicts
the label. We consider both sequential MNIST tasks, MNIST and permuted MNIST ( pMNIST). In
MNIST, the pixels are processed in scanline order. In pMNIST the pixels are processed in a ﬁxed
random order.
Our baseline consists of an LSTM with 100 hidden units, with a softmax classiﬁer to produce a
prediction from the ﬁnal hidden state. We use orthogonal initialization for all weight matrices,
except for the hidden-to-hidden weight matrix which we initialize to be the identity matrix, as this
yields better generalization performance on this task for both models. The model is trained using
RMSProp (Tieleman & Hinton, 2012) with learning rate of 103and0:9momentum. We apply
gradient clipping at 1 to avoid exploding gradients.

Recurrent Batch Normalization

Layer Normalization


gradient clipping at 1 to avoid exploding gradients.
The in-order MNIST task poses a unique problem for our model: the input for the ﬁrst hundred or so
timesteps is constant across examples since the upper pixels are almost always black. This causes the
variance of the hidden states to be exactly zero for a long period of time. Normalizing these zerovariance activations involves dividing zero by a small number at many timesteps, which does not
affect the forward-propagated activations but causes the back-propagated gradient to explode. We
work around this by adding Gaussian noise to the initial hidden states. Although the normalization
ampliﬁes the noise to signal level, we ﬁnd that it does not hurt performance compared to datadependent ways of initializing the hidden states.
Model MNIST pMNIST
TANH-RNN (Le et al., 2015) 35.0 35.0
iRNN (Le et al., 2015) 97.0 82.0
uRNN (Arjovsky et al., 2015) 95.1 91.4
sTANH-RNN (Zhang et al., 2016) 98.1 94.0
LSTM (ours) 98.9 90.2
BN-LSTM (ours) 99.0 95.4

Recurrent Batch Normalization

Layer Normalization


LSTM (ours) 98.9 90.2
BN-LSTM (ours) 99.0 95.4
Table 1: Accuracy obtained on the test set for the pixel by pixel MNIST classiﬁcation tasks
In Figure 2 we show the validation accuracy while training for both LSTM and batch-normalized
LSTM (BN-LSTM). BN-LSTM converges faster than LSTM on both tasks. Additionally, we observe that BN-LSTM generalizes signiﬁcantly better on pMNIST. It has been highlighted in Arjovsky et al. (2015) that pMNIST contains many longer term dependencies across pixels than in
the original pixel ordering, where a lot of structure is local. A recurrent network therefore needs to
5
Published as a conference paper at ICLR 2017
Model Penn Treebank
LSTM (Graves, 2013) 1.262
HF-MRNN (Mikolov et al., 2012) 1.41
Norm-stabilized LSTM (Krueger & Memisevic, 2016) 1.39
ME n-gram (Mikolov et al., 2012) 1.37
LSTM (ours) 1.38
BN-LSTM (ours) 1.32
Zoneout (Krueger et al., 2016) 1.27

Recurrent Batch Normalization

Layer Normalization


LSTM (ours) 1.38
BN-LSTM (ours) 1.32
Zoneout (Krueger et al., 2016) 1.27
HM-LSTM (Chung et al., 2016) 1.24
HyperNetworks (Ha et al., 2016) 1.22
Table 2: Bits-per-character on the Penn Treebank test sequence.
characterize dependencies across varying time scales in order to solve this task. Our results suggest
that BN-LSTM is better able to capture these long-term dependencies.
Table 1 reports the test set accuracy of the early stop model for LSTM and BN-LSTM using the population statistics. Recurrent batch normalization leads to a better test score, especially for pMNIST
where models have to leverage long-term temporal depencies. In addition, Table 1 shows that our
batch-normalized LSTM achieves state of the art on both MNIST and pMNIST.
5.2 C HARACTER -LEVEL PENN TREEBANK
We evaluate our model on the task of character-level language modeling on the Penn Treebank
corpus (Marcus et al., 1993) according to the train/valid/test partition of Mikolov et al. (2012). For
training, we segment the training sequence into examples of length 100. The training sequence does

Recurrent Batch Normalization

Layer Normalization


training, we segment the training sequence into examples of length 100. The training sequence does
not cleanly divide by 100, so for each epoch we randomly crop a subsequence that does and segment
that instead.
Our baseline is an LSTM with 1000 units, trained to predict the next character using a softmax
classiﬁer on the hidden state ht. We use stochastic gradient descent on minibatches of size 64, with
gradient clipping at 1.0 and step rule determined by Adam (Kingma & Ba, 2014) with learning rate
0.002. We use orthogonal initialization for all weight matrices. The setup for the batch-normalized
LSTM is the same in all respects except for the introduction of batch normalization as detailed in 3.
We show the learning curves in Figure 3(a). BN-LSTM converges faster and generalizes better than
the LSTM baseline. Figure 3(b) shows the generalization of our model to longer sequences. We
observe that using the population statistics improves generalization performance, which conﬁrms
that repeating the last population statistic (cf. Section 3) is a viable strategy. In table 2 we report
the performance of our best models (early-stopped on validation performance) on the Penn Treebank

Recurrent Batch Normalization

Layer Normalization


the performance of our best models (early-stopped on validation performance) on the Penn Treebank
test sequence. Follow up works havd since improved the state of the art (Krueger et al., 2016; Chung
et al., 2016; Ha et al., 2016).
5.3 T EXT8
We evaluate our model on a second character-level language modeling task on the much larger
text8 dataset (Mahoney, 2009). This dataset is derived from Wikipedia and consists of a sequence
of 100M characters including only alphabetical characters and spaces. We follow Mikolov et al.
(2012); Zhang et al. (2016) and use the ﬁrst 90M characters for training, the next 5M for validation
and the ﬁnal 5M characters for testing. We train on nonoverlapping sequences of length 180.
Both our baseline and batch-normalized models are LSTMs with 2000 units, trained to predict the
next character using a softmax classiﬁer on the hidden state ht. We use stochastic gradient descent
on minibatches of size 128, with gradient clipping at 1.0 and step rule determined by Adam (Kingma
& Ba, 2014) with learning rate 0.001. All weight matrices were initialized to be orthogonal.
6
Published as a conference paper at ICLR 2017

Recurrent Batch Normalization

Layer Normalization


& Ba, 2014) with learning rate 0.001. All weight matrices were initialized to be orthogonal.
6
Published as a conference paper at ICLR 2017
We early-stop on validation performance and report the test performance of the resulting model in
table 3. We observe that BN-LSTM obtains a signiﬁcant performance improvement over the LSTM
baseline. Chung et al. (2016) has since improved on our performance.
Model text8
td-LSTM (Zhang et al., 2016) 1.63
HF-MRNN (Mikolov et al., 2012) 1.54
skipping RNN (Pachitariu & Sahani, 2013) 1.48
LSTM (ours) 1.43
BN-LSTM (ours) 1.36
HM-LSTM (Chung et al., 2016) 1.29
Table 3: Bits-per-character on the text8 test sequence.
5.4 T EACHING MACHINES TO READ AND COMPREHEND
Recently, Hermann et al. (2015) introduced a set of challenging benchmarks for natural language
processing, along with neural network architectures to address them. The tasks involve reading

Recurrent Batch Normalization

Layer Normalization


Recently, Hermann et al. (2015) introduced a set of challenging benchmarks for natural language
processing, along with neural network architectures to address them. The tasks involve reading
real news articles and answering questions about their content. Their principal model, the Attentive Reader, is a recurrent neural network that invokes an attention mechanism to locate relevant
information in the document. Such models are notoriously hard to optimize and yet increasingly
popular.
To demonstrate the generality and practical applicability of our proposal, we apply batch normalization in the Attentive Reader model and show that this drastically improves training.
We evaluate several variants. The ﬁrst variant, referred to as BN-LSTM, consists of the vanilla Attentive Reader model with the LSTM simply replaced by our BN-LSTM reparameterization. The
second variant, termed BN-everywhere, is exactly like the ﬁrst, except that we also introduce batch
normalization into the attention computations, normalizing each term going into the tanh nonlinearities.
Our third variant, BN-e*, is like BN-everywhere, but improved to more carefully handle variablelength sequences. Throughout this experiment we followed the common practice of padding each
batch of variable-length data with zeros. However, this biases the batch mean and variance of xt

Recurrent Batch Normalization

Layer Normalization


batch of variable-length data with zeros. However, this biases the batch mean and variance of xt
toward zero. We address this effect using sequencewise normalization of the inputs as proposed
by Laurent et al. (2016); Amodei et al. (2015). That is, we share statistics over time for normalization
0 2000 4000 6000 8000 10000 12000 14000 16000
training steps1.41.61.82.02.22.4bits per characterLSTM
BN-LSTM
(a) Performance in bits-per-character on length100 subsequences of the Penn Treebank validation
sequence during training.
100 200 300 400 500 600 700 800 900 1000
sequence length1.321.341.361.381.401.421.441.46mean bits per characterLSTM
BN-LSTM, population statistics
BN-LSTM, batch statistics(b) Generalization to longer subsequences of Penn
Treebank using population statistics. The subsequences are taken from the test sequence.
Figure 3: Penn Treebank evaluation
7
Published as a conference paper at ICLR 2017
0 100 200 300 400 500 600 700 800
training steps (thousands)0.00.20.40.60.81.0error rateLSTM train
BN-LSTM train
BN-everywhere train
BN-e* train

Recurrent Batch Normalization

Layer Normalization


BN-LSTM train
BN-everywhere train
BN-e* train
BN-e** train
LSTM valid
BN-LSTM valid
BN-everywhere valid
BN-e* valid
BN-e** valid
(a) Error rate on the validation set for the Attentive Reader models on a variant of the CNN QA
task (Hermann et al., 2015). As detailed in Appendix C, the theoretical lower bound on the error
rate on this task is 43%.
0 50 100 150 200 250 300 350 400
training steps (thousands)0.10.20.30.40.50.60.70.80.91.0error rateLSTM train
BN-e** train
LSTM valid
BN-e** valid(b) Error rate on the validation set on the full CNN
QA task from Hermann et al. (2015).
Figure 4: Training curves on the CNN question-answering tasks.
of the input terms Wxxt, but notfor the recurrent terms Whhtor the cell output ct. Doing so avoids
many issues involving degenerate statistics due to input sequence padding.
Our fourth and ﬁnal variant BN-e** is like BN-e* but bidirectional. The main difﬁculty in adapting

Recurrent Batch Normalization

Layer Normalization


Our fourth and ﬁnal variant BN-e** is like BN-e* but bidirectional. The main difﬁculty in adapting
to bidirectional models also involves padding. Padding poses no problem as long as it is properly
ignored (by not updating the hidden states based on padded regions of the input). However to
perform the reverse application of a bidirectional model, it is common to simply reverse the padded
sequences, thus moving the padding to the front. This causes similar problems as were observed
on the sequential MNIST task (Section 5.1): the hidden states will not diverge during the initial
timesteps and hence their variance will be severely underestimated. To get around this, we reverse
only the unpadded portion of the input sequences and leave the padding in place.
See Appendix C for hyperparameters and task details.
Figure 4(a) shows the learning curves for the different variants of the attentive reader. BN-LSTM
trains dramatically faster than the LSTM baseline. BN-everywhere in turn shows a signiﬁcant improvement over BN-LSTM. In addition, both BN-LSTM and BN-everywhere show a generalization

Recurrent Batch Normalization

Layer Normalization


beneﬁt over the baseline. The validation curves have minima of 50.3%, 49.5% and 50.0% for the
baseline, BN-LSTM and BN-everywhere respectively. We emphasize that these results were obtained without any tweaking – all we did was to introduce batch normalization.
BN-e* and BN-e** converge faster yet, and reach lower minima: 47.1% and 43.9% respectively.
Model CNN valid CNN test
Attentive Reader (Hermann et al., 2015) 38.4 37.0
LSTM (ours) 45.5 45.0
BN-e** (ours) 37.9 36.3
Table 4: Error rates on the CNN question-answering task Hermann et al. (2015).
We train and evaluate our best model, BN-e**, on the full task from (Hermann et al., 2015). On
this dataset we had to reduce the number of hidden units to 120 to avoid severe overﬁtting. Training
curves for BN-e** and a vanilla LSTM are shown in Figure 4(b). Table 4 reports performances of
the early-stopped models.
8
Published as a conference paper at ICLR 2017
6 C ONCLUSION

Recurrent Batch Normalization

Layer Normalization


the early-stopped models.
8
Published as a conference paper at ICLR 2017
6 C ONCLUSION
Contrary to previous ﬁndings by Laurent et al. (2016); Amodei et al. (2015), we have demonstrated
that batch-normalizing the hidden states of recurrent neural networks greatly improves optimization. Indeed, doing so yields beneﬁts similar to those of batch normalization in feed-forward neural
networks: our proposed BN-LSTM trains faster and generalizes better on a variety of tasks including language modeling and question-answering. We have argued that proper initialization of
the batch normalization parameters is crucial, and suggest that previous difﬁculties (Laurent et al.,
2016; Amodei et al., 2015) were due in large part to improper initialization. Finally, we have shown
our model to apply to complex settings involving variable-length data, bidirectionality and highly
nonlinear attention mechanisms.
ACKNOWLEDGEMENTS
The authors would like to acknowledge the following agencies for research funding and computing
support: the Nuance Foundation, Samsung, NSERC, Calcul Québec, Compute Canada, the Canada
Research Chairs and CIFAR. Experiments were carried out using the Theano (Team et al., 2016) and

Recurrent Batch Normalization

Layer Normalization


Research Chairs and CIFAR. Experiments were carried out using the Theano (Team et al., 2016) and
the Blocks and Fuel (van Merriënboer et al., 2015) libraries for scientiﬁc computing. We thank David
Krueger, Saizheng Zhang, Ishmael Belghazi and Yoshua Bengio for discussions and suggestions.
REFERENCES
D. Amodei et al. Deep speech 2: End-to-end speech recognition in english and mandarin.
arXiv:1512.02595 , 2015.
M. Arjovsky, A. Shah, and Y . Bengio. Unitary evolution recurrent neural networks.
arXiv:1511.06464 , 2015.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450 ,
2016.
D. Bahdanau, K. Cho, and Y . Bengio. Neural machine translation by jointly learning to align and
translate. ICLR , 2015.
Y . Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is
difﬁcult. Neural Networks, IEEE Transactions on , 1994.

Recurrent Batch Normalization

Layer Normalization


difﬁcult. Neural Networks, IEEE Transactions on , 1994.
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio.
Learning phrase representations using rnn encoder-decoder for statistical machine translation.
arXiv:1406.1078 , 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv:1609.01704 , 2016.
A. Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850 , 2013.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv:1609.09106 , 2016.
K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom.
Teaching machines to read and comprehend. In NIPS , 2015.
S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Master’s thesis , 1991.

Recurrent Batch Normalization

Layer Normalization


S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Master’s thesis , 1991.
S. Hochreiter and J Schmidhuber. Long short-term memory. Neural computation , 1997.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. abs/1502.03167, 2015.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 , 2014.
D Krueger and R. Memisevic. Regularizing rnns by stabilizing activations. ICLR , 2016.
9
Published as a conference paper at ICLR 2017
David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Aaron Courville. Zoneout:
Regularizing rnns by randomly preserving hidden activations. arXiv:1606.01305 , 2016.
C. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural
networks. ICASSP , 2016.

Recurrent Batch Normalization

Layer Normalization


C. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural
networks. ICASSP , 2016.
Quoc V Le, N. Jaitly, and G. Hinton. A simple way to initialize recurrent networks of rectiﬁed linear
units. arXiv:1504.00941 , 2015.
Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural
networks and visual cortex. arXiv:1604.03640 , 2016.
M. Mahoney. Large text compression benchmark. 2009.
M. P. Marcus, M. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english:
The penn treebank. Comput. Linguist. , 1993.
J. Martens and I. Sutskever. Learning recurrent neural networks with hessian-free optimization. In
ICML , 2011.
T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky. Subword language
modeling with neural networks. preprint , 2012.
Yann Ollivier. Persistent contextual neural networks for learning symbolic data sequences. CoRR ,

Recurrent Batch Normalization

Layer Normalization


modeling with neural networks. preprint , 2012.
Yann Ollivier. Persistent contextual neural networks for learning symbolic data sequences. CoRR ,
abs/1306.0514, 2013.
Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are they needed? arXiv:1301.5650 , 2013.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difﬁculty of training recurrent neural
networks. arXiv:1211.5063 , 2012.
H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood
function. Journal of statistical planning and inference , 2000.
The Theano Development Team et al. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints , abs/1605.02688, May 2016.
T. Tieleman and G. Hinton. Lecture 6.5—RmsProp: Divide the gradient by a running average of its
recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.

Recurrent Batch Normalization

Layer Normalization


recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Bart van Merriënboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David WardeFarley, Jan Chorowski, and Yoshua Bengio. Blocks and fuel: Frameworks for deep learning.
CoRR , abs/1506.00619, 2015. URL http://arxiv.org/abs/1506.00619 .
K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y . Bengio. Show, attend and
tell: Neural image caption generation with visual attention. arXiv:1502.03044 , 2015.
L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by
exploiting temporal structure. In ICCV , 2015.
S. Zhang, Y . Wu, T. Che, Z. Lin, R. Memisevic, R. Salakhutdinov, and Y . Bengio. Architectural
complexity measures of recurrent neural networks. arXiv:1602.08210 , 2016.
10
Published as a conference paper at ICLR 2017

Recurrent Batch Normalization

Layer Normalization


complexity measures of recurrent neural networks. arXiv:1602.08210 , 2016.
10
Published as a conference paper at ICLR 2017
A C ONVERGENCE OF POPULATION STATISTICS
0.20
0.15
0.10
0.05
0.000.050.100.150.20mean of recurrent term
0.20
0.15
0.10
0.05
0.000.050.100.150.20mean of cell state
0 10 20 30 40 50
time steps0.00.51.01.52.0variance of recurrent term
0 10 20 30 40 50
time steps0.0010.0020.0030.0040.0050.0060.0070.0080.009variance of cell state
Figure 5: Convergence of population statistics to stationary distributions on the Penn Treebank task.
The horizontal axis denotes RNN time. Each curve corresponds to a single hidden unit. Only a
random subset of units is shown. See Section 3 for discussion.
B S ENSITIVITY TO INITIALIZATION OF
In Section 4 we investigated the effect of initial
on gradient ﬂow. To show the practical implications of this, we performed several experiments on the pMNIST and Penn Treebank benchmarks.
The resulting performances are shown in Figure 6.

Recurrent Batch Normalization

Layer Normalization


The resulting performances are shown in Figure 6.
ThepMNIST training curves conﬁrm that higher initial values of
are detrimental to the optimization of the model. For the Penn Treebank task however, the effect is gone.
We believe this is explained by the difference in the nature of the two tasks. For pMNIST, the model
absorbs the input sequence and only at the end of the sequence does it make a prediction on which it
receives feedback. Learning from this feedback requires propagating the gradient all the way back
through the sequence.
In the Penn Treebank task on the other hand, the model makes a prediction at each timestep. At
each step of the backward pass, a fresh learning signal is added to the backpropagated gradient.
Essentially, the model is able to get off the ground by picking up short-term dependencies. This fails
onpMNIST wich is dominated by long-term dependencies (Arjovsky et al., 2015).
C T EACHING MACHINES TO READ AND COMPREHEND : TASK SETUP
We evaluate the models on the question answering task using the CNN corpus (Hermann et al.,
2015), with placeholders for the named entities. We follow a similar preprocessing pipeline as Hermann et al. (2015). During training, we randomly sample the examples with replacement and shufﬂe

Recurrent Batch Normalization

Layer Normalization


the order of the placeholders in each text inside the minibatch. We use a vocabulary of 65829 words.
We deviate from Hermann et al. (2015) in order to save computation: we use only the 4 most relevant
sentences from the description, as identiﬁed by a string matching procedure. Both the training and
validation sets are preprocessed in this way. Due to imprecision this heuristic sometimes strips the
11
Published as a conference paper at ICLR 2017
0 10000 20000 30000 40000 50000
training steps0.00.51.01.52.02.5cross entropyPermuted MNIST train
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
0 10000 20000 30000 40000 50000
training steps0.00.51.01.52.02.5cross entropyPermuted MNIST valid
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
0 5000 10000 15000
training steps0.800.850.900.951.001.051.10bits per characterPTB train
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70

Recurrent Batch Normalization

Layer Normalization


gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
0 5000 10000 15000
training steps1.001.021.041.061.081.10bits per characterPTB valid
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
Figure 6: Training curves on pMNIST and Penn Treebank for various initializations of
.
answers from the passage, putting an upper bound of 57% on the validation accuracy that can be
achieved.
For the reported performances, the ﬁrst three models (LSTM, BN-LSTM and BN-everywhere) are
trained using the exact same hyperparameters, which were chosen because they work well for the
baseline. The hidden state is composed of 240 units. We use stochastic gradient descent on minibatches of size 64, with gradient clipping at 10 and step rule determined by Adam (Kingma & Ba,
2014) with learning rate 8105.
For BN-e* and BN-e**, we use the same hyperparameters except that we reduce the learning rate to
8104and the minibatch size to 40.

Recurrent Batch Normalization

Layer Normalization


8104and the minibatch size to 40.
D H YPERPARAMETER SEARCHES
Table 5 reports hyperparameter values that were tried in the experiments.
(a) MNIST and pMNIST
Learning rate: 1e-2, 1e-3, 1e-4
RMSProp momentum: 0.5, 0.9
Hidden state size: 100, 200, 400
Initial
: 1e-1, 3e-1, 5e-1, 7e-1, 1.0(b) Penn Treebank
Learning rate: 1e-1, 1e-2, 2e-2, 1e-3
Hidden state size: 800, 1000, 1200, 1500, 2000
Batch size: 32, 64, 100, 128
Initial
: 1e-1, 3e-1, 5e-1, 7e-1, 1.0
(c) Text8
Learning rate: 1e-1, 1e-2, 1e-3
Hidden state size: 500, 1000, 2000, 4000(d) Attentive Reader
Learning rate: 8e-3, 8e-4, 8e-5, 8e-6
Hidden state size: 60, 120, 240, 280
Table 5: Hyperparameter values that have been explored in the experiments.

Recurrent Batch Normalization

Layer Normalization


Hidden state size: 60, 120, 240, 280
Table 5: Hyperparameter values that have been explored in the experiments.
For MNIST and pMNIST, the hyperparameters were varied independently. For Penn Treebank, we
performed a full grid search on learning rate and hidden state size, and later performed a sensitivity
12
Published as a conference paper at ICLR 2017
analysis on the batch size and initial
. For the text8 task and the experiments with the Attentive
Reader, we carried out a grid search on the learning rate and hidden state size.
The same values were tried for both the baseline and our BN-LSTM. In each case, our reported
results are those of the model with the best validation performance.
13

Recurrent Batch Normalization

Layer Normalization


Convolutional Random Walk Networks for Semantic Image Segmentation
Gedas Bertasius1, Lorenzo Torresani2, Stella X. Yu3, Jianbo Shi1
1University of Pennsylvania,2Dartmouth College,3UC Berkeley ICSI
{gberta,jshi}@seas.upenn.edu lt@dartmouth.edu stella.yu@berkeley.edu
Abstract
Most current semantic segmentation methods rely on
fully convolutional networks (FCNs). However, their use of
large receptive ﬁelds and many pooling layers cause low
spatial resolution inside the deep layers. This leads to
predictions with poor localization around the boundaries.
Prior work has attempted to address this issue by postprocessing predictions with CRFs or MRFs. But such models often fail to capture semantic relationships between objects, which causes spatially disjoint predictions. To overcome these problems, recent methods integrated CRFs or
MRFs into an FCN framework. The downside of these new
models is that they have much higher complexity than traditional FCNs, which renders training and testing more challenging.
In this work we introduce a simple, yet effective Convolutional Random Walk Network (RWN) that addresses the
issues of poor boundary localization and spatially fragmented predictions with very little increase in model complexity. Our proposed RWN jointly optimizes the objectives

Summary:
Most current semantic segmentation methods rely on fully convolutional
networks (FCNs). However, their use of large receptive fields and many pooling
layers cause low spatial resolution inside the deep layers. This leads to
predictions with poor localization around the boundaries. Prior work has
attempted to address this issue by post-processing predictions with CRFs or
MRFs. But such models often fail to capture semantic relationships between
objects, which causes spatially disjoint predictions. To overcome these
problems, recent methods integrated CRFs or MRFs into an FCN framework. The
downside of these new models is that they have much higher complexity than
traditional FCNs, which renders training and testing more challenging.
  In this work we introduce a simple, yet effective Convolutional Random Walk
Network (RWN) that addresses the issues of poor boundary localization and
spatially fragmented predictions with very little increase in model complexity.
Our proposed RWN jointly optimizes the objectives of pixelwise affinity and
semantic segmentation. It combines these two objectives via a novel random walk
layer that enforces consistent spatial grouping in the deep layers of the
network. Our RWN is implemented using standard convolution and matrix
multiplication. This allows an easy integration into existing FCN frameworks
and it enables end-to-end training of the whole network via standard
back-propagation. Our implementation of RWN requires just $131$ additional
parameters compared to the traditional FCNs, and yet it consistently produces
an improvement over the FCNs on semantic segmentation and scene labeling.

Gedas Bertasius
Lorenzo Torresani
Stella X. Yu
Jianbo Shi


issues of poor boundary localization and spatially fragmented predictions with very little increase in model complexity. Our proposed RWN jointly optimizes the objectives
of pixelwise afﬁnity and semantic segmentation. It combines these two objectives via a novel random walk layer
that enforces consistent spatial grouping in the deep layers
of the network. Our RWN is implemented using standard
convolution and matrix multiplication. This allows an easy
integration into existing FCN frameworks and it enables
end-to-end training of the whole network via standard backpropagation. Our implementation of RWN requires just 131
additional parameters compared to the traditional FCNs,
and yet it consistently produces an improvement over the
FCNs on semantic segmentation and scene labeling.
1. Introduction
Fully convolutional networks (FCNs) were ﬁrst introduced in [20] where they were shown to yield signiﬁcant
improvements in semantic image segmentation. Adopting the FCN approach, many subsequent methods have
achieved even better performance [6, 27, 8, 12, 17, 6, 27, 8,
19, 23]. However, traditional FCN-based methods tend to
suffer from several limitations. Large receptive ﬁelds in the
Input
 DeepLab [6]
 DeepLab-CRF [6]


suffer from several limitations. Large receptive ﬁelds in the
Input
 DeepLab [6]
 DeepLab-CRF [6]
Figure 1: Examples illustrating shortcomings of prior semantic segmentation methods. Segments produced by
FCNs are poorly localized around object boundaries, while
Dense-CRF produce spatially-disjoint object segments.
convolutional layers and the presence of pooling layers lead
to low spatial resolution in the deepest FCN layers. As a result, their predicted segments tend to be blobby and lack ﬁne
object boundary details. We report in Fig. 1 some examples
illustrating typical poor localization of objects in the outputs
of FCNs. Recently, Chen at al. [6] addressed this issue by
applying a Dense-CRF post-processing step [15] on top of
coarse FCN segmentations. However, such approaches often fail to accurately capture semantic relationships between
objects and lead to spatially fragmented segmentations (see
an example in the last column of Fig. 1).
To address these problems several recent methods integrated CRFs or MRFs directly into the FCN framework [27, 19, 17, 5]. However, these new models typically involve (1) a large number of parameters, (2) complex loss functions requiring specialized model training or
(3) recurrent layers, which make training and testing more


(3) recurrent layers, which make training and testing more
complicated. We summarize the most prominent of these
approaches and their model complexities in Table 1.
We note that we do not claim that using complex loss
functions always makes the model overly complex and too
difﬁcult to use. If a complex loss is integrated into an FCN
framework such that the FCN can still be trained in a standard fashion, and produce better results than using standard losses, then such a model is beneﬁcial. However, in
the context of prior segmentation methods [27, 19, 17, 5],
such complex losses often require: 1) modifying the network structure (casting CNN into an RNN) [27, 5], or 2)
using a complicated multi-stage learning scheme, where
different layers are optimized during a different training
stage [19, 17]. Due to such complex training procedures,
1arXiv:1605.07681v3  [cs.CV]  8 May 2017
[6] [5] [23] [27] [19] RWN
requires post-processing? 3777 7 7
uses complex loss? 7773 3 7
requires recurrent layers? 7373 7 7
model size (in MB) 79 79 961 514 >1000 79


uses complex loss? 7773 3 7
requires recurrent layers? 7373 7 7
model size (in MB) 79 79 961 514 >1000 79
Table 1: Summary of recent semantic segmentation models. For each model, we report whether it requires: (1) CRF
post-processing, (2) complex loss functions, or (3) recurrent layers. We also list the size of the model (using the
size of Caffe [13] models in MB). We note that unlike prior
methods, our RWN does not need post-processing, it is implemented using standard layers and loss functions, and it
also has a compact model.
which are adapted for speciﬁc tasks and datasets, these
models can be quite difﬁcult to adapt for new tasks and
datasets, which is disadvantageous.
Inspired by random walk methods [21, 3, 24], in this
work we introduce a simple, yet effective alternative to traditional FCNs: a Convolutional Random Walk Network
(RWN) that combines the strengths of FCNs and random walk methods. Our model addresses the issues of
(1) poor localization around the boundaries suffered by
FCNs and (2) spatially disjoint segments produced by dense
CRFs. Additionally, unlike recent semantic segmentation


(1) poor localization around the boundaries suffered by
FCNs and (2) spatially disjoint segments produced by dense
CRFs. Additionally, unlike recent semantic segmentation
approaches [23, 27, 19, 17], our RWN does so without signiﬁcantly increasing the complexity of the model.
Our proposed RWN jointly optimizes (1) pixelwise afﬁnity and (2) semantic segmentation learning objectives that
are linked via a novel random walk layer, which enforces
spatial consistency in the deepest layers of the network. The
random walk layer is implemented via matrix multiplication. As a result, RWN seamlessly integrates both afﬁnity
and segmentation branches, and can be jointly trained endto-end via standard back-propagation with minimal modiﬁcations to the existing FCN framework. Additionally, our
implementation of RWN requires only 131additional parameters. Thus, the effective complexity of our model is
the same as the complexity of traditional FCNs (see Table 1). We compare our approach to several variants of the
DeepLab semantic segmentation system [6, 7], and show
that our proposed RWN consistently produces better performance over these baselines for the tasks of semantic segmentation and scene labeling.
2. Related Work
The recent introduction of fully convolutional networks


that our proposed RWN consistently produces better performance over these baselines for the tasks of semantic segmentation and scene labeling.
2. Related Work
The recent introduction of fully convolutional networks
(FCNs) [20] has led to remarkable advances in semantic
segmentation. However, due to the large receptive ﬁelds
and many pooling layers, segments predicted by FCNs tend
to be blobby and lack ﬁne object boundary details. Recently
there have been several attempts to address these problems.
These approaches can be divided into several groups.The work in [4, 6, 25, 14, 4] used FCN predictions as
unary potentials in a separate globalization model that reﬁnes the segments using similarity cues based on regions
or boundaries. One disadvantage of these methods is that
the learning of the unary potentials and the training of the
globalization model are completely disjoint. As a result,
these methods often fail to capture semantic relationships
between objects, which produces segmentation results that
are spatially disjoint (see the right side of Fig. 1).
To address these issues, several recent methods [27, 19,
17] have proposed to integrate a CRF or a MRF into the network, thus enabling end-to-end training of the joint model.


17] have proposed to integrate a CRF or a MRF into the network, thus enabling end-to-end training of the joint model.
However, the merging of these two models leads to a dramatic increase in complexity and number of parameters.
For instance, the method in [27], requires to cast the original
FCN into a Recurrent Neural Network (RNN), which renders the model much bigger in size (see Table 1). A recent
method [5] jointly predicts boundaries and segmentations,
and then combines them using a recurrent layer, which also
requires complex modiﬁcations to the existing FCN framework.
The work in [19] proposes to use local convolutional layers, which leads to a signiﬁcantly larger number of parameters. Similarly, the method in [17] proposes to model unary
and pairwise potentials by separate multi-scale branches.
This leads to a network with at least twice as many parameters as the traditional FCN and a much more complex multistage training procedure.
In addition to the above methods, it is worth mentioning
deconvolutional networks [23, 12], which use deconvolution and unpooling layers to recover ﬁne object details from
the coarse FCN predictions. However, in order to effectively recover ﬁne details one must employ nearly as many


the coarse FCN predictions. However, in order to effectively recover ﬁne details one must employ nearly as many
deconvolutional layers as the number of convolutional layers, which yields a large growth in number of parameters
(see Table 1).
Unlike these prior methods, our implementation of RWN
needs only 131additional parameters over the base FCN.
These additional parameters represent only 0:0008% of the
total number of parameters in the network. In addition, our
RWN uses standard convolution and matrix multiplication.
Thus, it does not need to incorporate complicated loss functions or new complex layers [27, 5, 19, 17]. Finally, unlike the methods in [4, 6, 25, 14] that predict and reﬁne the
segmentations disjointly, our RWN model jointly optimizes
pixel afﬁnity and semantic segmentation in an end-to-end
fashion. Our experiments show that this leads to spatially
smoother segmentations.
3. Background
Random Graph Walks. Random walks are one of
the most widely known and used methods in graph theory [21]. Most notably, the concept of random walks led to
FC8 ActivationsDeepLab FCN Architecture
RGB ImageSparse Pairwise L1 Distances    Sparse Pixel    Similarity Matrix1 x 1 x k conv
   Ground Truth   Pixel Similarities


RGB ImageSparse Pairwise L1 Distances    Sparse Pixel    Similarity Matrix1 x 1 x k conv
   Ground Truth   Pixel Similarities
 Exp
   Ground Truth     Segmentation
      Segmentation            Random Walk Layerˆy=AfAffinity Learning BranchPairwise L1  Distances
n⇥n⇥mRow-Normalization:n⇥n⇥3
f2Rn2⇥mA2Rn2⇥n2W2Rn2⇥n2F2Rn2⇥n2⇥kFigure 2: The architecture of our proposed Random Walk Network (RWN) (best viewed in color). Our RWN consists
of two branches: (1) one branch devoted to the segmentation predictions , and (2) another branch predicting pixel-level
afﬁnities. These two branches are then merged via a novel random walk layer that encourages spatially smooth segmentation
predictions. The entire RWN is jointly optimized end-to-end via a standard back-propagation algorithm.
the development of PageRank [24] and Personalized PageRank [3], which are widely used for many applications. Let
G= (V;E)denote an undirected graph with a set of verticesVand a set of edges E. Then a random walk in


G= (V;E)denote an undirected graph with a set of verticesVand a set of edges E. Then a random walk in
such graph can be characterized by the transition probabilities between its vertices. Let Wbe a symmetric nn
afﬁnity matrix, where ndenotes the number of nodes in
the graph and where Wij2[0;1]denotes how similar the
nodesiandjare. In the context of a semantic segmentation problem, each pixel in the image can be viewed as
a separate node in the graph, where the similarity between
two nodes can be evaluated according to some metric (e.g.
color or texture similarity etc). Then let Dindicate a diagonalnnmatrix, which stores the degree values for each
node:Dii=Pn
j=1Wijfor alljexcepti=j. Then, we can
express our random walk transition matrix as A=D1W.
Given this setup, we want to model how the information
in the graph spreads if we start at a particular node, and
perform a random walk in this graph. Let ytbe an1vector denoting a node distribution at time t. In the context of
the PageRank algorithm, ytmay indicate the rank estimates
associated with each of the nWeb pages at time t. Then,


the PageRank algorithm, ytmay indicate the rank estimates
associated with each of the nWeb pages at time t. Then,
according to the random walk theory, we can spread the
rank information in the graph by performing a one-step random walk. This process can be expressed as yt+1=Ayt,
whereyt+1denotes a newly obtained rank distribution after
one random walk step, the matrix Acontains the random
walk transition probabilities, and ytis the rank distribution
at time step t. Thus, we can observe that the information
among the nodes can be diffused, by simply multiplying the
random walk transition probability matrix A, with the rank
distribution ytat a particular time t. This process can be
repeated multiple times, until convergence is reached. For a
more detailed survey please see [21, 24].
Difference from MRF/CRF Approaches. CRFs andMRFs have been widely used in structured prediction problems [16]. Recently, CRFs and MRFs have also been integrated into the fully convolutional network framework for
semantic segmentation [27, 19, 17]. We want to stress that
while the goals of CRF/MRF and random walk methods are
the same (i.e. to globally propagate information in the graph
structures), the mechanism to achieve this objective is very


the same (i.e. to globally propagate information in the graph
structures), the mechanism to achieve this objective is very
different in these two approaches. While MRFs and CRFs
typically employ graphs with a ﬁxed grid structure (e.g., one
where each node is connected to its four closest neighbors),
random walk methods are much more ﬂexible and can implement any arbitrary graph structure via the afﬁnity matrix
speciﬁcation. Thus, since our proposed RWN is based on
random walks, it can employ any arbitrary graph structure,
which can be beneﬁcial as different problems may require
different graph structures.
Additionally, to globally propagate information among
the nodes, MRFs and CRFs need to employ approximate
inference techniques, because exact inference tends to be
intractable in graphs with a grid structure. Integrating such
approximate inference techniques into the steps of FCN
training and prediction can be challenging and may require
lots of domain-speciﬁc modiﬁcations. In comparison, random walk methods globally propagate information among
the nodes via a simple matrix multiplication. Not only is
the matrix multiplication efﬁcient and exact, but it is also
easy to integrate into the traditional FCN framework for
both training and prediction schemes. Additionally, due to


the matrix multiplication efﬁcient and exact, but it is also
easy to integrate into the traditional FCN framework for
both training and prediction schemes. Additionally, due to
the use of standard convolution and matrix multiplication
operations, our RWN can be trivially trained via standard
back-propagation in an end-to-end fashion.
4. Convolutional Random Walk Networks
In this work, our goal is to integrate a random walk process into the FCN architecture to encourage coherent semantic segmentation among pixels that are similar to each
other. Such a process introduces an explicit grouping mechanism, which should be beneﬁcial in addressing the issues
of (1) poor localization around the boundaries, and (2) spatially fragmented segmentations.
A schematic illustration of our proposed RWN architecture is presented in Fig. 2. Our RWN is a network composed of two branches: (1) one branch that predicts semantic segmentation potentials, and (2) another branch devoted
to predicting pixel-level afﬁnities. These two branches are
merged via a novel random walk layer that encourages spatially coherent semantic segmentation. The entire RWN can
be jointly optimized end-to-end. We now describe each of
the components of the RWN architecture in more detail.
4.1. Semantic Segmentation Branch
For the semantic segmentation branch, we present results


the components of the RWN architecture in more detail.
4.1. Semantic Segmentation Branch
For the semantic segmentation branch, we present results
for several variants of the DeepLab segmentation systems,
including DeepLab-LargeFOV [6], DeepLab-attention [7],
and DeepLab-v2, which is one of the top performing segmentation systems. DeepLab-largeFOV is a fully convolutional adaptation of the VGG [26] architecture, which contains 16convolutional layers. DeepLab-attention [7], is a
multi-scale VGG based network, for which each multi-scale
branch focuses on a speciﬁc part of the image. Finally,
DeepLab-v2 is a multi-scale network based on the residual network [11] implementation. We note that even though
we use a DeepLab architecture in our experiments, other architectures such as [2] and many others could be integrated
into our framework.
4.2. Pixel-Level Afﬁnity Branch
To learn the pairwise pixel-level afﬁnities, we employ a
separate afﬁnity learning branch with its own learning objective (See Fig. 2). The afﬁnity branch is connected with
the inputnn3RGB image, and low-level conv1_1and


the inputnn3RGB image, and low-level conv1_1and
conv1_2layers. The feature maps corresponding to these
layers arennin width and height but they have a different number of channels ( 3;64;and64respectively). Let k
be the total number of afﬁnity learning parameters (in our
casek= 3 + 64 + 64 = 131 ). Then, let Fbe a sparse
n2n2kmatrix that stores L1distances between each
pixel and all of its neighbors within a radius R, according
to each channel. Note that the distances are notsummed
up across the kchannels, but instead they are computed and
stored separately for each channel. The resulting matrix F
is then used as an input to the afﬁnity branch, as shown in
Figure 2.
The afﬁnity branch consists of a 11kconvolutional
layer and an exponential layer. The output of the exponential layer is then attached to the Euclidean loss layer and is
optimized to predict the ground truth pixel afﬁnities, which
are obtained from the original semantic segmentation anInput
 DeepLab_v2
 RWN_v2
Figure 3: A ﬁgure illustrating the segmentation results of


are obtained from the original semantic segmentation anInput
 DeepLab_v2
 RWN_v2
Figure 3: A ﬁgure illustrating the segmentation results of
our RWN and the DeepLab-v2 network. Note that RWN
produced segmentations are spatially smoother and produce
less false positive predictions than the DeepLab-v2 system.
notations. Speciﬁcally, we set the ground truth afﬁnity between two pixels to 1if the pixels share the same semantic
label and have distance less than Rfrom each other. Note
thatF, which is used as an input to the afﬁnity branch, is a
sparse matrix, as only a small fraction of all the entries in F
are populated with non-zero values. The rest of the entries
are ignored during the computation.
Also note that we only use features from RGB, conv1_1
andconv1_2layers, because they are not affected by pooling, and thus, preserve the original spatial resolution. We
also experimented with using features from deeper FCN
layers such as fc6, and fc7. However, we observed that
features from deeper layers are highly correlated to the
predicted semantic segmentation unary potentials, which
causes redundancy and little improvement in the segmentation performance. We also experimented with using more


features from deeper layers are highly correlated to the
predicted semantic segmentation unary potentials, which
causes redundancy and little improvement in the segmentation performance. We also experimented with using more
than one convolutional layer in the afﬁnity learning branch,
but observed that additional layers provide negligible improvements in accuracy.
4.3. Random Walk Layer
To integrate the semantic segmentation potentials and
our learned pixel-level afﬁnities, we introduce a novel random walk layer, which propagates the semantic segmentation information based on the learned afﬁnities. The random
walk layer is connected to the two bottom layers: (1) the fc8
layer containing the semantic segmentation potentials, and
(2) the afﬁnity layer that outputs a sparse n2n2random
walk transition matrix A. Then, let fdenote the activation values from the fc8layer, reshaped to the dimensions
ofn2m, wheren2refers to the number of pixels, and
mis the number of object classes in the dataset. A single
random walk layer implements one step of the random walk
Method aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mean overall


metric. From the results, we observe that our proposed RWN consistently outperforms DeepLab-LargeFOV , DeepLabattention, and DeepLab-v2 baselines.
process, which can be performed as ^y=Af, where ^yindicates the diffused segmentation predictions, and Adenotes
the random walk transition matrix.
The random walk layer is then attached to the softmax
loss layer, and is optimized to predict ground truth semantic
segmentations. One of the advantages of our proposed random walk layer is that it is implemented as a matrix multiplication, which makes it possible to back-propagate the
gradients to both (1) the afﬁnity branch and (2) the segmentation branch. Let the softmax-loss gradient be an n2m
matrix@L
@^y, wheren2is the number of pixels in the fc8layer,
andmis the number of predicted object classes. Then the
gradients, which are back-propagated to the semantic segmentation branch are computed as@L
@f=AT@L
@^y, whereAT
is the transposed random walk transition matrix. Also, the
gradients, that are back-propagated to the afﬁnity branch
are computed as@L
@A=@L


gradients, that are back-propagated to the afﬁnity branch
are computed as@L
@A=@L
@^yfT, wherefTis amn2matrix
that contains transposed activation values from the fc8layer
of the segmentation branch. We note that@L
@Ais asparse
n2n2matrix, which means that the above matrix multiplication only considers the pixel pairs that correspond to
the non-zero entries in the random walk transition matrix A.
4.4. Random Walk Prediction at Testing
In the previous subsection, we mentioned that the prediction in the random walk layer can be done via a simple
matrix multiplication operation ^y=Af, whereAdenotes
the random walk transition matrix, and fdepicts the activation values from the fc8layer. Typically, we want to apply
multiple random walk steps until convergence is reached.
However, we also do not want to deviate too much from our
initial segmentation predictions, in case the random walk
transition matrix is not perfectly accurate, which is a reasonable expectation. Thus, our prediction scheme needs to
balance two effects: (1) propagating the segmentation information across the nodes using a random walk transition
matrix, and (2) not deviating too much from the initial segmentation.


balance two effects: (1) propagating the segmentation information across the nodes using a random walk transition
matrix, and (2) not deviating too much from the initial segmentation.
This tradeoff between the two quantities is very similar to the idea behind MRF and CRF models, which try to
minimize an energy formed by unary and pairwise terms.
However, as discussed earlier, MRF and CRF methods tend
to use 1) grid structure graphs and 2) various approximateinference techniques to propagate segmentation information globally. In comparison, our random walk approach
is advantageous because it can use 1) any arbitrary graph
structure and 2) an exact matrix multiplication operation to
achieve the same goal.
Let us ﬁrst denote our segmentation prediction after t+1
random walk steps as ^yt+1. Then our general prediction
scheme can be written as:
^yt+1=A^yt+ (1)f (1)
wheredenotes a parameter [0;1]that controls the
tradeoff between (1) diffusing segmentation information
along the connections of a random walk transition matrix
and (2) not deviating too much from initial segmentation
values (i.e. the outputs of the last layer of the FCN). Let us
now initialize ^y0to contain the output values from the fc8


values (i.e. the outputs of the last layer of the FCN). Let us
now initialize ^y0to contain the output values from the fc8
layer, which we denoted with f. Then we can write our prediction equation by substituting the recurrent expressions:
^yt+1= (A)t+1f+ (1)tX
i=0(A)if (2)
Now, because we want to apply our random walk procedure until convergence we set t=1. Then, because
our random walk transition matrix is stochastic we know
thatlimt!1(A)t+1= 0 . Furthermore, we can write
St=Pt
i=0Ai=I+A+A2+:::+At, whereIis
an identity matrix, and where Stdenotes a partial sum of
random walk transitions until iteration t. We can then write
StASt=IAt+1, which implies:
(IA)St=IAt+1(3)
From our previous derivation, we already know that
limt!1(A)t+1= 0, which implies that
S1= (IA)1(4)
Thus, our ﬁnal prediction equation, which corresponds
to applying repeated random walk steps until convergence,


S1= (IA)1(4)
Thus, our ﬁnal prediction equation, which corresponds
to applying repeated random walk steps until convergence,
can be written as
^y1= (IA)1f (5)
Input
 DeepLab_v2-CRF
 RWN_v2
Figure 4: Comparison of segmentation results produced by
our RWN versus the DeepLab-v2-CRF system. It can be noticed that, despite not using any post-processing steps, our
RWN predicts ﬁne object details (e.g., bike wheels or plane
wings) more accurately than DeepLab-v2-CRF, which fails
to capture some these object parts.
In practice, the random walk transition matrix Ais pretty
large, and inverting it is impractical. To deal with this problem, we shrink the matrix (IA)using a simple and efﬁcient technique presented in [1], and then invert it to compute the ﬁnal segmentation. In the experimental section, we
show that such a prediction scheme produces solid results
and is still pretty efﬁcient ( 1 second per image). We also
note that we use this prediction scheme only during testing. During training we employ a scheme that uses a single


note that we use this prediction scheme only during testing. During training we employ a scheme that uses a single
random walk step (but with a much larger radius), which is
faster. We explain this procedure in the next subsection.
4.5. Implementation Details
We jointly train our RWN in an end-to-end fashion for
2000 iterations, with a learning rate of 105,0:9momentum, the weight decay of 5105, and 15samples per batch.
For the RWN model, we set the tradeoff parameter to
0:01. During testing we set the random walk connectivity
radiusR= 5 and apply the random walk procedure until convergence. However, during training we set R= 40 ,
and apply a single random walk step. This training strategy
works well because increasing the radius size eliminates the
need for multiple random walk steps, which speeds up the
training. However, using R= 5 and applying an inﬁnite
number of random walk steps until convergence still yields
slightly better results (see study in 5.4), so we use it during testing. For all of our experiments, we use a Caffe library [13]. During training, we also employ data augmentation techniques such as cropping, and mirroring.
5. Experimental Results


5. Experimental Results
In this section, we present our results for semantic segmentation on the SBD [10] dataset, which contains objects
and their per-pixel labels for 20Pascal VOC classes (excluding the background class). We also include scene label-Method mean IOU overall IOU
DeepLab-largeFOV-CRF 75.7 77.7
RWN-largeFOV 75.8 77.9
DeepLab-attention-CRF 79.9 81.6
RWN-attention 79.9 81.5
DeepLab-v2-CRF 81.9 84.2
RWN-v2 82.1 84.3
DeepLab-DT 76.6 78.7
RWN 76.7 78.8
Table 3: Quantitative comparison between our RWN model
and several variants of the DeepLab system that use a dense
CRF or a domain-transfer (DT) ﬁlter for post-processing.
These results suggest that our RWN acts as an effective
globalization scheme, since it produces results that are
similar or even better than the results achieved by postprocessing the DeepLab outputs with a CRF or DT.
ing results on the commonly used Stanford background [9]
and Sift Flow [18] datasets. We evaluate our segmentation results on these tasks using the standard metric of the


ing results on the commonly used Stanford background [9]
and Sift Flow [18] datasets. We evaluate our segmentation results on these tasks using the standard metric of the
intersection-over-union (IOU) averaged per pixels across all
the classes from each dataset. We also include the classagnostic overall pixel intersection-over-union score, which
measures the per-pixel IOU across all classes.
We experiment with several variants of the DeepLab
system [6, 7] as our main baselines throughout our experiments: DeepLab-LargeFOV [6], DeepLab-attention [7],
and DeepLab-v2.
Our evaluations provide evidence for four conclusions:
In subsections 5.1, 5.2, we demonstrate that our proposed RWN outperforms DeepLab baselines for both
semantic segmentation, and scene labeling tasks.
In subsection 5.1, we demonstrate that, compared to
the dense CRF approaches, RWN predicts segmentations that are spatially smoother.
In Subsection 5.3, we show that our approach is more
efﬁcient than the denseCRF inference.
Finally, in Subsection, 5.4, we demonstrate that our
random walk layer is beneﬁcial and that our model is
ﬂexible to use different graph structures.


random walk layer is beneﬁcial and that our model is
ﬂexible to use different graph structures.
5.1. Semantic Segmentation Task
Standard Evaluation. In Table 2, we present semantic
segmentation results on the Pascal SBD dataset [10], which
contains 8055 training and 2857 testing images. These results indicate that RWN consistently outperforms all three
of the DeepLab baselines. In Figure 3, we also compare
qualitative segmentation results of a DeepLab-v2 network
and our RWN model. We note that the RWN segmentations contain fewer false positive predictions and are also
spatially smoother across the object regions.
0 2 4 6 8 10 12 14 16 18 200.20.250.30.350.40.450.50.55
Trimap Width (in Pixels)Pixel Classification Error (%)Localization Around the Boundaries Error

DeepLab
RWNFigure 5: Localization error around the object boundaries
within a trimap. Compared to the DeepLab system (blue),
our RWN (red) achieves lower segmentation error around
object boundaries for all trimap widths.
Furthermore, in Table 3, we present experiments where
we compare RWN with models using dense CRFs [15] to
post-process the predictions of DeepLab systems. We also


Furthermore, in Table 3, we present experiments where
we compare RWN with models using dense CRFs [15] to
post-process the predictions of DeepLab systems. We also
include DeepLab-DT [5], which uses domain-transfer ﬁltering to reﬁne the segmentations inside an FCN. Based on
these results, we observe that, despite not using any postprocessing, our RWN produces results similar to or even
better than the DeepLab models employing post-processing.
These results indicate that RWN can be used as a globalization mechanism to ensure spatial coherence in semantic
segmentation predictions. In Figure 4 we present qualitative results where we compare the ﬁnal segmentation predictions of RWN and the DeepLab-v2-CRF system. Based
on these qualitative results, we observe that RWN captures
more accurately the ﬁne details of the objects, such as the
bike wheels, or plane wings. The DeepLab-v2-CRF system
misses some of these object parts.
Localization Around the Boundaries. Earlier we
claimed that due to the use of large receptive ﬁelds and
many pooling layers, FCNs tend to produce blobby segmentations that lack ﬁne object boundary details. We want


many pooling layers, FCNs tend to produce blobby segmentations that lack ﬁne object boundary details. We want
to show that our RWN produces more accurate segmentations around object boundaries the traditional FCNs. Thus,
adopting the practice from [15], we evaluate segmentation
accuracy around object boundaries. We do so by counting
the relative number of misclassiﬁed pixels within a narrow
band (“trimap”) surrounding the ground truth object boundaries. We present these results in Figure 5. The results show
that RWN achieves higher segmentation accuracy than the
DeepLab (DL) system for all trimap widths considered in
this test.
Spatial Smoothness. We also argued that applying the
dense CRF [15] as a post-processing technique often leads
to spatially fragmented segmentations (see the right side of
Fig. 1). How can we evaluate whether a given method produces spatially smooth or spatially fragmented segmenta-Method MF AP
DeepLab-largeFOV-CRF 0.676 0.457
RWN-largeFOV 0.703 0.494
DeepLab-attention-CRF 0.722 0.521
RWN-attention 0.747 0.556
DeepLab-v2-CRF 0.763 0.584


RWN-attention 0.747 0.556
DeepLab-v2-CRF 0.763 0.584
RWN-v2 0.773 0.595
Table 4: Quantitative comparison of spatial segmentation
smoothness. We extract the boundaries from the predicted
segmentations and evaluate them against ground truth object boundaries using max F-score (MF) and average precision (AP) metrics. These results suggest that RWN segmentations are spatially smoother than the DeepLab-CRF
segmentations across all baselines.
tions? Intuitively, spatially fragmented segmentations will
produce many false boundaries that do not correspond to
actual object boundaries. Thus, to test the spatial smoothness of a given segmentation, we extract the boundaries
from the segmentation and then compare these boundaries
against ground truth object boundaries using the standard
maximum F-score (MF) and average precision (AP) metrics, as done in the popular BSDS benchmark [22]. We perform this experiment on the Pascal SBD dataset and present
these results in Table 4. We can see that the boundaries extracted from the RWN segmentations yield better MF and
AP results compared to the boundaries extracted from the
different variants of the DeepLab-CRF system. Thus, these
results suggest that RWN produces spatially smoother segmentations than DeepLab-CRF.
5.2. Scene Labeling


results suggest that RWN produces spatially smoother segmentations than DeepLab-CRF.
5.2. Scene Labeling
We also tested our RWN on the task of scene labeling
using two popular datasets: Stanford Background [9] and
Sift Flow [18]. Stanford Background is a relatively small
dataset for scene labeling. It contains 715images, which
we randomly split into 600training images and 115testing images. In contrast, the Sift Flow dataset contains 2489
training examples and 201testing images. For all of our
experiments, we use the DeepLab-largeFOV [6] architecture since it is smaller and more efﬁcient to train and test.
To evaluate scene labeling results, we use the overall IOU
evaluation metric which is a commonly used metric for this
task. In Table 5, we present our scene labeling results on
both of these datasets. Our results indicate that our RWN
method outperforms the DeepLab baseline by 2:57%, and
2:54% on these two datasets, respectively.
5.3. Runtime Comparisons
We also include the runtime comparison of our RWN approach versus the denseCRF inference. We note that using a single core of a 2:7GHz Intel Core i7 processor, the
denseCRF inference requires 3.301 seconds per image on
Input Image
Iteration 0Iteration 50


denseCRF inference requires 3.301 seconds per image on
Input Image
Iteration 0Iteration 50
Figure 6: A ﬁgure illustrating how the probability predictions change as we apply more random walk steps. Note
that the RWN predictions become more reﬁned and better localized around the object boundaries as more random
walk steps are applied.
average on a Pascal SBD dataset. In comparison, a single
iteration of a random walk, which is simply a sparse matrix
multiplication, takes 0.032 seconds on average on the same
Pascal SBD dataset. A DeepLab_v2 post-processed with
denseCRF achieves 81:9%IOU score on this same Pascal
SBD dataset. In comparison, RWN_v2 with a single random walk iteration and with R=40 (radius) achieves 82:2%
IOU, which is both more accurate and more than 100times
more efﬁcient than the denseCRF inference.
5.4. Ablation Experiments
Optimal Number of Random Walk Steps. In Figure 7,
we illustrate how the IOU accuracy changes when we use a
different number of random walk steps. We observe that the
segmentation accuracy keeps increasing as we apply more
random walk steps, and that it reaches its peak performance
when the random walk process converges, which indicates


segmentation accuracy keeps increasing as we apply more
random walk steps, and that it reaches its peak performance
when the random walk process converges, which indicates
the effectiveness of our random walk step procedure. In
Figure 6, we also illustrate how the predicted object segmentation probabilities change as we apply more random
walk steps. We observe that the object boundaries become
much better localized as more iterations of random walk are
applied.
Radius Size. To analyze the effect of a radius size in
the RWN architecture, we test alternative versions of our
model with different radii sizes. Our results indicate, that
the RWN model produces similar results with different radii
in the interval of R > 3andR < 20if the random walk
step process is applied until convergence. We also note
that, if we select R= 40 , and apply a random walk step
only once , we can achieve the segmentation accuracy of
75:5%and77:6%according to the two evaluation metrics,
0 5 10 15 20 25 30 40 50 60 inf0.7350.740.7450.750.7550.760.765
Number of Random Walk StepsIOU AccuracyAccuracy versus the Number of Random Walk Steps


Number of Random Walk StepsIOU AccuracyAccuracy versus the Number of Random Walk Steps

Mean−class IOUFigure 7: IOU accuracy as a function of the number of random walk steps. From this plot we observe that the segmentation accuracy keeps improving as we apply more random walk steps and that it reaches its peak when the random
walk process converges.
DeepLab-largeFOV RWN-largeFOV
Stanford-BG 65.74 68.31
Sift-Flow 67.31 69.85
Table 5: Scene labeling results on the Stanford Background
and Sift-Flow datasets measured according to the overall
IOU evaluation metric. We use a DeepLab-largeFOV network as base model, and show that our RWN yields better
results on both of these scene labeling datasets.
respectively. In comparison, choosing R= 5 and applying random walk until convergence yields the accuracies of
75:8%and77:9%, which is slightly better. However, note
that selecting R= 40 , and applying multiple random walk
steps does not yield any improvement in segmentation accuracy. These experiments show the ﬂexibility of our model
compared to the MRF or CRF models, which typically use
graphs with a ﬁxed grid structure. Our model has the ability
to use different graph structures depending on the problem.
6. Conclusion


graphs with a ﬁxed grid structure. Our model has the ability
to use different graph structures depending on the problem.
6. Conclusion
In this work, we introduced Random Walk Networks
(RWNs), and showed that, compared to traditional fully
convolutional networks (FCNs), they produce improved accuracy for the same model complexity. Our RWN addresses
the issues of 1) poor localization around the segmentation
boundaries and 2) spatially disjoint segmentations. Additionally, our implementation of RWN uses only 131additional learnable parameters ( 0:0008% of the original number of the parameters in the network) and it can be easily
integrated into the standard FCN learning framework for a
joint end-to-end training. Finally, RWN provides a more
efﬁcient alternative to the denseCRF approaches.
Our future work includes experimenting with alternative RWN architectures and applying RWN to new domains
such as language processing or speech recognition.
References
[1] Pablo Arbelaez, J. Pont-Tuset, Jon Barron, F. Marqués, and
Jitendra Malik. Multiscale combinatorial grouping. In Computer Vision and Pattern Recognition (CVPR) , 2014. 6
[2] Vijay Badrinarayanan, Ankur Handa, and Roberto Cipolla.


[2] Vijay Badrinarayanan, Ankur Handa, and Roberto Cipolla.
Segnet: A deep convolutional encoder-decoder architecture
for robust semantic pixel-wise labelling. arXiv preprint
arXiv:1505.07293 , 2015. 4
[3] Bahman Bahmani, Abdur Chowdhury, and Ashish Goel. Fast
incremental and personalized pagerank. Proc. VLDB Endow., 4(3):173–184, December 2010. 2, 3
[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural ﬁelds. In The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 2
[5] Liang-Chieh Chen, Jonathan T. Barron, George Papandreou,
Kevin Murphy, and Alan L. Yuille. Semantic image segmentation with task-speciﬁc edge detection using cnns and a
discriminatively trained domain transform. CVPR , 2016. 1,
2, 7
[6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,


2, 7
[6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,
Kevin Murphy, and Alan L. Yuille. Semantic image segmentation with deep convolutional nets and fully. In ICLR , 2015.
1, 2, 4, 6, 7
[7] Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and
Alan L. Yuille. Attention to scale: Scale-aware semantic
image segmentation. CVPR , 2016. 2, 4, 6
[8] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting
bounding boxes to supervise convolutional networks for semantic segmentation. In The IEEE International Conference
on Computer Vision (ICCV) , December 2015. 1
[9] S. Gould, R. Fulton, and D. Koller. Decomposing a scene
into geometric and semantically consistent regions. In Proceedings of the International Conference on Computer Vision (ICCV) , 2009. 6, 7
[10] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev,
Subhransu Maji, and Jitendra Malik. Semantic contours from
inverse detectors. In International Conference on Computer
Vision (ICCV) , 2011. 6