C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV , 2014.
Kyunghyun Cho, Bart Van Merri ¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger
Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical
machine translation. EMNLP , 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.
ICLR , 2015.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 , 2014.
D. Kingma and J. L. Ba. Adam: a method for stochastic optimization. ICLR , 2014. arXiv:1412.6980.
Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings.
CVPR , 2016.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman,

Layer Normalization

Batch Normalized Recurrent


CVPR , 2016.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman,
and Phil Blunsom. Teaching machines to read and comprehend. In NIPS , 2015.
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja
Fidler. Skip-thought vectors. In NIPS , 2015.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efﬁcient estimation of word representations in
vector space. arXiv preprint arXiv:1301.3781 , 2013.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja
Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading
books. In ICCV , 2015.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli.
Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through

Layer Normalization

Summary:
Training state-of-the-art, deep neural networks is computationally expensive.
One way to reduce the training time is to normalize the activities of the
neurons. A recently introduced technique called batch normalization uses the
distribution of the summed input to a neuron over a mini-batch of training
cases to compute a mean and variance which are then used to normalize the
summed input to that neuron on each training case. This significantly reduces
the training time in feed-forward neural networks. However, the effect of batch
normalization is dependent on the mini-batch size and it is not obvious how to
apply it to recurrent neural networks. In this paper, we transpose batch
normalization into layer normalization by computing the mean and variance used
for normalization from all of the summed inputs to the neurons in a layer on a
single training case. Like batch normalization, we also give each neuron its
own adaptive bias and gain which are applied after the normalization but before
the non-linearity. Unlike batch normalization, layer normalization performs
exactly the same computation at training and test times. It is also
straightforward to apply to recurrent neural networks by computing the
normalization statistics separately at each time step. Layer normalization is
very effective at stabilizing the hidden state dynamics in recurrent networks.
Empirically, we show that layer normalization can substantially reduce the
training time compared with previously published techniques.

Jimmy Lei Ba
Jamie Ryan Kiros
Geoffrey E. Hinton

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Batch Normalized Recurrent


Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through
semantic relatedness and textual entailment. SemEval-2014 , 2014.
11
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect
to rating scales. In ACL, pages 115–124, 2005.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM
SIGKDD international conference on Knowledge discovery and data mining , 2004.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based
on minimum cuts. In ACL, 2004.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation , 2005.
K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. DRAW: a recurrent neural network for image generation.
arXiv:1502.04623, 2015.
Hugo Larochelle and Iain Murray. The neural autoregressive distribution estimator. In AISTATS , volume 6,
page 622, 2011.
Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten

Layer Normalization

Batch Normalized Recurrent


page 622, 2011.
Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten
text on a whiteboard. In ICDAR , 2005.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013.
12
Supplementary Material
Application of layer normalization to each experiment
This section describes how layer normalization is applied to each of the papers’ experiments. For
notation convenience, we deﬁne layer normalization as a function mapping LN:RD!RDwith
two set of adaptive parameters, gains and biases :
LN(z;;) =(z)
+; (15)
=1
DDX
i=1zi;  =vuut1
DDX
i=1(zi)2; (16)
where,ziis theithelement of the vector z.
Teaching machines to read and comprehend and handwriting sequence generation
The basic LSTM equations used for these experiment are given by:
0
B@ft
it
ot
gt1
CA=Whht1+Wxxt+b (17)

Layer Normalization

Batch Normalized Recurrent


0
B@ft
it
ot
gt1
CA=Whht1+Wxxt+b (17)
ct=(ft)ct1+(it)tanh(gt) (18)
ht=(ot)tanh(ct) (19)
The version that incorporates layer normalization is modiﬁed as follows:
0
B@ft
it
ot
gt1
CA=LN(Whht1;1;1) +LN(Wxxt;2;2) +b (20)
ct=(ft)ct1+(it)tanh(gt) (21)
ht=(ot)tanh(LN(ct;3;3)) (22)
where i;iare the additive and multiplicative parameters, respectively. Each iis initialized to a
vector of zeros and each iis initialized to a vector of ones.
Order embeddings and skip-thoughts
These experiments utilize a variant of gated recurrent unit which is deﬁned as follows:

zt
rt
=Whht1+Wxxt (23)

Layer Normalization

Batch Normalized Recurrent


zt
rt
=Whht1+Wxxt (23)
^ht=tanh(Wxt+(rt)(Uht1)) (24)
ht= (1(zt))ht1+(zt)^ht (25)
Layer normalization is applied as follows:

zt
rt
=LN(Whht1;1;1) +LN(Wxxt;2;2) (26)
^ht=tanh(LN(Wxt;3;3) +(rt)LN(Uht1;4;4)) (27)
ht= (1(zt))ht1+(zt)^ht (28)
just as before, iis initialized to a vector of zeros and each iis initialized to a vector of ones.
13
Modeling binarized MNIST using DRAW
The layer norm is only applied to the output of the LSTM hidden states in this experiment:
The version that incorporates layer normalization is modiﬁed as follows:0
B@ft
it
ot
gt1
CA=Whht1+Wxxt+b (29)

Layer Normalization

Batch Normalized Recurrent


B@ft
it
ot
gt1
CA=Whht1+Wxxt+b (29)
ct=(ft)ct1+(it)tanh(gt) (30)
ht=(ot)tanh(LN(ct;;)) (31)
where ;are the additive and multiplicative parameters, respectively. is initialized to a vector
of zeros and is initialized to a vector of ones.
Learning the magnitude of incoming weights
We now compare how gradient descent updates changing magnitude of the equivalent weights between the normalized GLM and original parameterization. The magnitude of the weights are explicitly parameterized using the gain parameter in the normalized model. Assume there is a gradient
update that changes norm of the weight vectors by g. We can project the gradient updates to the
weight vector for the normal GLM. The KL metric, ie how much the gradient update changes the
model prediction, for the normalized model depends only on the magnitude of the prediction error.
Speciﬁcally,
under batch normalization:
ds2=1
2vec([0;0;g]>)>F(vec([W;b;g]>) vec([0;0;g]>) =1
2>
gE

Layer Normalization

Batch Normalized Recurrent


Layer Normalization

Batch Normalized Recurrent


g1
2E
xP(x)2
6664Cov(y1;y1jx)a2
1
kw1k2
2 Cov(y1;yHjx)a1aH
kw1k2kwHk2
.........
Cov(yH;y1jx)aHa1
kwHk2kw1k2 Cov(yH;yHjx)a2
H
kwHk2
23
7775g:(34)
Whereas, the KL metric in the standard GLM is related to its activities ai=w>
ix, that is depended
on both its current weights and input data. We project the gradient updates to the gain parameter gi
of theithneuron to its weight vector as giwi
kwik2in the standard GLM model:
1
2vec([giwi
kwik2;0;gjwj
kwik2;0]>)>F([w>
i;bi;w>
j;bj]>) vec([giwi
kwik2;0;gjwj
kwjk2;0]>)
=gigj
22E
xP(x)

Layer Normalization

Batch Normalized Recurrent


kwik2;0;gjwj
kwjk2;0]>)
=gigj
22E
xP(x)
Cov(yi;yjjx)aiaj
kwik2kwjk2
(35)
The batch normalized and layer normalized models are therefore more robust to the scaling of the
input and its parameters than the standard model.
14

Layer Normalization

Batch Normalized Recurrent


Published as a conference paper at ICLR 2017
HIERARCHICAL MULTISCALE
RECURRENT NEURAL NETWORKS
Junyoung Chung, Sungjin Ahn & Yoshua Bengio
Département d’informatique et de recherche opérationnelle
Université de Montréal
{junyoung.chung,sungjin.ahn,yoshua.bengio}@umontreal.ca
ABSTRACT
Learning both hierarchical and temporal representation has been among the longstanding challenges of recurrent neural networks. Multiscale recurrent neural
networks have been considered as a promising approach to resolve this issue, yet
there has been a lack of empirical evidence showing that this type of models can
actually capture the temporal dependencies by discovering the latent hierarchical
structure of the sequence. In this paper, we propose a novel multiscale approach,
called the hierarchical multiscale recurrent neural network, that can capture the
latent hierarchical structure in the sequence by encoding the temporal dependencies
with different timescales using a novel update mechanism. We show some evidence
that the proposed model can discover underlying hierarchical structure in the
sequences without using explicit boundary information. We evaluate our proposed
model on character-level language modelling and handwriting sequence generation.
1 I NTRODUCTION
One of the key principles of learning in deep neural networks as well as in the human brain is to obtain

Hierarchical Multiscale Recurrent Neural Networks

Summary:
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.

Junyoung Chung
Sungjin Ahn
Yoshua Bengio


1 I NTRODUCTION
One of the key principles of learning in deep neural networks as well as in the human brain is to obtain
a hierarchical representation with increasing levels of abstraction (Bengio, 2009; LeCun et al., 2015;
Schmidhuber, 2015). A stack of representation layers, learned from the data in a way to optimize
the target task, make deep neural networks entertain advantages such as generalization to unseen
examples (Hoffman et al., 2013), sharing learned knowledge among multiple tasks, and discovering
disentangling factors of variation (Kingma & Welling, 2013). The remarkable recent successes of
the deep convolutional neural networks are particularly based on this ability to learn hierarchical
representation for spatial data (Krizhevsky et al., 2012). For modelling temporal data, the recent
resurgence of recurrent neural networks (RNN) has led to remarkable advances (Mikolov et al., 2010;
Graves, 2013; Cho et al., 2014; Sutskever et al., 2014; Vinyals et al., 2015). However, unlike the
spatial data, learning both hierarchical and temporal representation has been among the long-standing
challenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many


spatial data, learning both hierarchical and temporal representation has been among the long-standing
challenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many
temporal data (Schmidhuber, 1991; Mozer, 1993; El Hihi & Bengio, 1995; Lin et al., 1996; Koutník
et al., 2014).
A promising approach to model such hierarchical and temporal representation is the multiscale
RNNs (Schmidhuber, 1992; El Hihi & Bengio, 1995; Koutník et al., 2014). Based on the observation
that high-level abstraction changes slowly with temporal coherency while low-level abstraction
has quickly changing features sensitive to the precise local timing (El Hihi & Bengio, 1995), the
multiscale RNNs group hidden units into multiple modules of different timescales. In addition to
the fact that the architecture ﬁts naturally to the latent hierarchical structures in many temporal data,
the multiscale approach provides the following advantages that resolve some inherent problems
of standard RNNs: (a) computational efﬁciency obtained by updating the high-level layers less
frequently, (b) efﬁciently delivering long-term dependencies with fewer updates at the high-level


frequently, (b) efﬁciently delivering long-term dependencies with fewer updates at the high-level
layers, which mitigates the vanishing gradient problem, (c) ﬂexible resource allocation (e.g., more
hidden units to the higher layers that focus on modelling long-term dependencies and less hidden
units to the lower layers which are in charge of learning short-term dependencies). In addition, the
learned latent hierarchical structures can provide useful information to other downstream tasks such
Yoshua Bengio is CIFAR Senior Fellow.
1arXiv:1609.01704v7  [cs.LG]  9 Mar 2017
Published as a conference paper at ICLR 2017
as module structures in computer program learning, sub-task structures in hierarchical reinforcement
learning, and story segments in video understanding.
There have been various approaches to implementing the multiscale RNNs. The most popular
approach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutník et al., 2014;
Bahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the
data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016). However, considering the fact that


data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016). However, considering the fact that
non-stationarity is prevalent in temporal data, and that many entities of abstraction such as words
and sentences are in variable length, we claim that it is important for an RNN to dynamically adapt
its timescales to the particulars of the input entities of various length. While this is trivial if the
hierarchical boundary structure is provided (Sordoni et al., 2015), it has been a challenge for an RNN
to discover the latent hierarchical structure in temporal data without explicit boundary information.
In this paper, we propose a novel multiscale RNN model, which can learn the hierarchical multiscale
structure from temporal data without explicit boundary information. This model, called a hierarchical
multiscale recurrent neural network (HM-RNN), does not assign ﬁxed update rates, but adaptively
determines proper update times corresponding to different abstraction levels of the layers. We ﬁnd
that this model tends to learn ﬁne timescales for low-level layers and coarse timescales for high-level
layers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is
turned on only at the time steps where a segment of the corresponding abstraction level is completely


layers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is
turned on only at the time steps where a segment of the corresponding abstraction level is completely
processed. Otherwise, i.e., during the within segment processing, it stays turned off. Using the
hierarchical boundary states, we implement three operations, UPDATE, COPY and FLUSH, and
choose one of them at each time step. The UPDATE operation is similar to the usual update rule of
the long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997), except that it is executed
sparsely according to the detected boundaries. The COPY operation simply copies the cell and hidden
states of the previous time step. Unlike the leaky integration of the LSTM or the Gated Recurrent
Unit (GRU) (Cho et al., 2014), the COPY operation retains the whole states without any loss of
information. The FLUSH operation is executed when a boundary is detected, where it ﬁrst ejects the
summarized representation of the current segment to the upper layer and then reinitializes the states
to start processing the next segment. Learning to select a proper operation at each time step and to
detect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We


to start processing the next segment. Learning to select a proper operation at each time step and to
detect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We
ﬁnd that the straight-through estimator (Hinton, 2012; Bengio et al., 2013; Courbariaux et al., 2016)
is efﬁcient for training this model containing discrete variables.
We evaluate our model on two tasks: character-level language modelling and handwriting sequence
generation. For the character-level language modelling, the HM-RNN achieves the state-of-the-art
results on the Text8 dataset, and comparable results to the state-of-the-art on the Penn Treebank
and Hutter Prize Wikipedia datasets. The HM-RNN also outperforms the standard RNN on the
handwriting sequence generation using the IAM-OnDB dataset. In addition, we demonstrate that the
hierarchical structure found by the HM-RNN is indeed very similar to the intrinsic structure observed
in the data. The contributions of this paper are:
We propose for the ﬁrst time an RNN model that can learn a latent hierarchical structure of
a sequence without using explicit boundary information.
We show that it is beneﬁcial to utilize the above structure through empirical evaluation.


a sequence without using explicit boundary information.
We show that it is beneﬁcial to utilize the above structure through empirical evaluation.
We show that the straight-through estimator is an efﬁcient way of training a model containing
discrete variables.
We propose the slope annealing trick to improve the training procedure based on the
straight-through estimator.
2 R ELATED WORK
Two notable early attempts inspiring our model are Schmidhuber (1992) and El Hihi & Bengio (1995).
In these works, it is advocated to stack multiple layers of RNNs in a decreasing order of update
frequency for computational and learning efﬁciency. In Schmidhuber (1992), the author shows a
model that can self-organize a hierarchical multiscale structure. Particularly in El Hihi & Bengio
(1995), the advantages of incorporating a priori knowledge, “ temporal dependencies are structured
hierarchically ", into the RNN architecture is studied. The authors propose an RNN architecture that
updates each layer with a ﬁxed but different rate, called a hierarchical RNN.
2
Published as a conference paper at ICLR 2017
LSTMs (Hochreiter & Schmidhuber, 1997) employ the multiscale update concept, where the hidden


2
Published as a conference paper at ICLR 2017
LSTMs (Hochreiter & Schmidhuber, 1997) employ the multiscale update concept, where the hidden
units have different forget and update rates and thus can operate with different timescales. However,
unlike our model, these timescales are not organized hierarchically. Although the LSTM has a selfloop for the gradients that helps to capture the long-term dependencies by mitigating the vanishing
gradient problem, in practice, it is still limited to a few hundred time steps due to the leaky integration
by which the contents to memorize for a long-term is gradually diluted at every time step. Also, the
model remains computationally expensive because it has to perform the update at every time step
for each unit. However, our model is less prone to these problems because it learns a hierarchical
structure such that, by design, high-level layers learn to perform less frequent updates than low-level
layers. We hypothesize that this property mitigates the vanishing gradient problem more efﬁciently
while also being computationally more efﬁcient.
A more recent model, the clockwork RNN (CW-RNN) (Koutník et al., 2014) extends the hierarchical


A more recent model, the clockwork RNN (CW-RNN) (Koutník et al., 2014) extends the hierarchical
RNN (El Hihi & Bengio, 1995) and the NARX RNN (Lin et al., 1996)1. The CW-RNN tries to
solve the issue of using soft timescales in the LSTM, by explicitly assigning hard timescales. In the
CW-RNN, hidden units are partitioned into several modules, and different timescales are assigned to
the modules such that a module iupdates its hidden units at every 2(i1)-th time step. The CW-RNN
is computationally more efﬁcient than the standard RNN including the LSTM since hidden units
are updated only at the assigned clock rates. However, ﬁnding proper timescales in the CW-RNN
remains as a challenge whereas our model learns the intrinsic timescales from the data. In the biscale
RNNs (Chung et al., 2016), the authors proposed to model layer-wise timescales adaptively by having
additional gating units, however this approach still relies on the softgating mechanism like LSTMs.
Other forms of Hierarchical RNN (HRNN) architectures have been proposed in the cases where


Other forms of Hierarchical RNN (HRNN) architectures have been proposed in the cases where
the explicit hierarchical boundary structure is provided. In Ling et al. (2015), after obtaining the
word boundary via tokenization, the HRNN architecture is used for neural machine translation by
modelling the characters and words using the ﬁrst and second RNN layers, respectively. A similar
HRNN architecture is also adopted in Sordoni et al. (2015) to model dialogue utterances. However,
in many cases, hierarchical boundary information is not explicitly observed or expensive to obtain.
Also, it is unclear how to deploy more layers than the number of boundary levels that is explicitly
observed in the data.
While the above models focus on online prediction problems, where a prediction needs to be made by
using only the past data, in some cases, predictions are made after observing the whole sequence. In
this setting, the input sequence can be regarded as 1-D spatial data, convolutional neural networks
with 1-D kernels are proposed in Kim (2014) and Kim et al. (2015) for language modelling and
sentence classiﬁcation. Also, in Chan et al. (2016) and Bahdanau et al. (2016), the authors proposed
to obtain high-level representation of the sequences of reduced length by repeatedly merging or
pooling the lower-level representation of the sequences.


to obtain high-level representation of the sequences of reduced length by repeatedly merging or
pooling the lower-level representation of the sequences.
Hierarchical RNN architectures have also been used to discover the segmentation structure in
sequences (Fernández et al., 2007; Kong et al., 2015). It is however different to our model in the
sense that they optimize the objective with explicit labels on the hierarchical segments while our
model discovers the intrinsic structure only from the sequences without segment label information.
The COPY operation used in our model can be related to Zoneout (Krueger et al., 2016) which
is a recurrent generalization of stochastic depth (Huang et al., 2016). In Zoneout, an identity
transformation is randomly applied to each hidden unit at each time step according to a Bernoulli
distribution. This results in occasional copy operations of the previous hidden states. While the focus
of Zoneout is to propose a regularization technique similar to dropout (Srivastava et al., 2014) (where
the regularization strength is controlled by a hyperparameter), our model learns (a) to dynamically
determine when to copy from the context inputs and (b) to discover the hierarchical multiscale
structure and representation. Although the main goal of our proposed model is not regularization, we
found that our model also shows very good generalization performance.


structure and representation. Although the main goal of our proposed model is not regularization, we
found that our model also shows very good generalization performance.
3
Published as a conference paper at ICLR 2017
h20h22h25h15h14h13h12h11h10x1x2x3x4x5word1word2phrase1
h20h21h22h23h24h25h15h14h13h12h11h10x1x2x3x4x5word1word2phrase1
(a) (b)
Figure 1: (a) The HRNN architecture, which requires the knowledge of the hierarchical boundaries.
(b) The HM-RNN architecture that discovers the hierarchical multiscale structure in the data.
3 H IERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS
3.1 M OTIVATION
To begin with, we provide an example of how a stacked RNN can model temporal data in an ideal
setting, i.e., when the hierarchy of segments is provided (Sordoni et al., 2015; Ling et al., 2015). In
Figure 1 (a), we depict a hierarchical RNN (HRNN) for language modelling with two layers: the ﬁrst
layer receives characters as inputs and generates word-level representations (C2W-RNN), and the


layer receives characters as inputs and generates word-level representations (C2W-RNN), and the
second layer takes the word-level representations as inputs and yields phrase-level representations
(W2P-RNN).
As shown, by means of the provided end-of-word labels, the C2W-RNN obtains word-level representation after processing the last character of each word and passes the word-level representation to the
W2P-RNN. Then, the W2P-RNN performs an update of the phrase-level representation. Note that the
hidden states of the W2P-RNN remains unchanged while all the characters of a word are processed by
the C2W-RNN. When the C2W-RNN starts to process the next word, its hidden states are reinitialized
using the latest hidden states of the W2P-RNN, which contain summarized representation of all the
words that have been processed by that time step, in that phrase.
From this simple example, we can see the advantages of having a hierarchical multiscale structure: (1)
as the W2P-RNN is updated at a much slower update rate than the C2W-RNN, a considerable amount
of computation can be saved, (2) gradients are backpropagated through a much smaller number of


of computation can be saved, (2) gradients are backpropagated through a much smaller number of
time steps, and (3) layer-wise capacity control becomes possible (e.g., use a smaller number of hidden
units in the ﬁrst layer which models short-term dependencies but whose updates are invoked much
more often).
Can an RNN discover such hierarchical multiscale structure without explicit hierarchical boundary
information? Considering the fact that the boundary information is difﬁcult to obtain (for example,
consider languages where words are not always cleanly separated by spaces or punctuation symbols,
and imperfect rules are used to separately perform segmentation) or usually not provided at all, this is
a legitimate problem. It gets worse when we consider higher-level concepts which we would like
the RNN to discover autonomously. In Section 2, we discussed the limitations of the existing RNN
models under this setting, which either have to update all units at every time step or use ﬁxed update
frequencies (El Hihi & Bengio, 1995; Koutník et al., 2014). Unfortunately, this kind of approach is
not well suited to the case where different segments in the hierarchical decomposition have different
lengths: for example, different words have different lengths, so a ﬁxed hierarchy would not update its


lengths: for example, different words have different lengths, so a ﬁxed hierarchy would not update its
upper-level units in synchrony with the natural boundaries in the data.
3.2 T HEPROPOSED MODEL
A key element of our model is the introduction of a parametrized boundary detector, which outputs
a binary value, in each layer of a stacked RNN, and learns when a segment should end in such
a way to optimize the overall target objective. Whenever the boundary detector is turned on at a
time step of layer `(i.e., when the boundary state is 1), the model considers this to be the end of a
1The acronym NARX stands for Non-linear Auto-Regressive model with eXogenous inputs.
4
Published as a conference paper at ICLR 2017
segment corresponding to the latent abstraction level of that layer (e.g., word or phrase) and feeds the
summarized representation of the detected segment into the upper layer ( `+ 1). Using the boundary
states, at each time step, each layer selects one of the following operations: UPDATE, COPY or
FLUSH. The selection is determined by (1) the boundary state of the current time step in the layer
belowz`1
tand (2) the boundary state of the previous time step in the same layer z`
t1.


belowz`1
tand (2) the boundary state of the previous time step in the same layer z`
t1.
In the following, we describe an HM-RNN based on the LSTM update rule. We call this model
a hierarchical multiscale LSTM (HM-LSTM). Consider an HM-LSTM model of Llayers (`=
1;:::;L ) which, at each layer `, performs the following update at time step t:
h`
t;c`
t;z`
t=f`
HM-LSTM (c`
t1;h`
t1;h`1
t;h`+1
t1;z`
t1;z`1
t): (1)
Here, handcdenote the hidden and cell states, respectively. The function f`
HM-LSTM is implemented
as follows. First, using the two boundary states z`
t1andz`1
t, the cell state is updated by:
c`
t=8
<
:f`
tc`
t1+i`
tg`
tifz`
t1= 0andz`1
t= 1(UPDATE)
c`


t1+i`
tg`
tifz`
t1= 0andz`1
t= 1(UPDATE)
c`
t1 ifz`
t1= 0andz`1
t= 0(COPY)
i`
tg`
t ifz`
t1= 1(FLUSH);(2)
and then the hidden state is obtained by:
h`
t=h`
t1 if COPY ,
o`
ttanh (c`
t)otherwise.(3)
Here, (f;i;o)are forget, input, output gates, and gis a cell proposal vector. Note that unlike the
LSTM, it is not necessary to compute these gates and cell proposal values at every time step. For
example, in the case of the COPY operation, we do not need to compute any of these values and thus
can save computations.
The COPY operation, which simply performs (c`
t;h`
t) (c`
t1;h`
t1), implements the observation
that an upper layer should keep its state unchanged until it receives the summarized input from
the lower layer. The UPDATE operation is performed to update the summary representation of the
layer`if the boundary z`1


the lower layer. The UPDATE operation is performed to update the summary representation of the
layer`if the boundary z`1
tis detected from the layer below but the boundary z`
t1was not found
at the previous time step. Hence, the UPDATE operation is executed sparsely unlike the standard
RNNs where it is executed at every time step, making it computationally inefﬁcient. If a boundary is
detected, the FLUSH operation is executed. The FLUSH operation consists of two sub-operations:
(a) EJECT to pass the current state to the upper layer and then (b) RESET to reinitialize the state
before starting to read a new segment. This operation implicitly forces the upper layer to absorb the
summary information of the lower layer segment, because otherwise it will be lost. Note that the
FLUSH operation is a hard reset in the sense that it completely erases all the previous states of the
same layer, which is different from the softreset or softforget operation in the GRU or LSTM.
Whenever needed (depending on the chosen operation), the gate values ( f`
t;i`
t;o`
t), the cell proposal
g`
t, and the pre-activation of the boundary detector ~z`
t2are then obtained by:
0
BBB@f`
t
i`


g`
t, and the pre-activation of the boundary detector ~z`
t2are then obtained by:
0
BBB@f`
t
i`
t
o`
t
g`
t
~z`
t1
CCCA=0
BBB@sigm
sigm
sigm
tanh
hard sigm1
CCCAfslice
srecurrent (`)
t +stop-down (`)
t +sbottom-up (`)
t +b(`)
; (4)
where
srecurrent (`)
t =U`
`h`
t1; (5)
stop-down (`)
t =z`
t1U`
`+1h`+1
t1; (6)
sbottom-up (`)
t =z`1
tW`
`1h`1
t: (7)
Here, we use Wj
i2R(4dim (h`)+1)dim (h`1);Uj
i2R(4dim (h`)+1)dim (h`)to denote state transition


i2R(4dim (h`)+1)dim (h`)to denote state transition
parameters from layer ito layerj, and b2R4dim (h`)+1is a bias term. In the last layer L, the
2~z`
tcan also be implemented as a function of h`
t, e.g., ~z`
t=hard sigm (Uh`
t).
5
Published as a conference paper at ICLR 2017
1z`t1z`t1W``1z`1tz`t1h`+1t1h`t1h`+1th`th`1tU``+1U``W`+1`z`t
ˆxt+1g1tg2tg3th1th2th3thet
Figure 2: Left: The gating mechanism of the HM-RNN. Right: The output module when L= 3.
top-down connection is ignored, and we use h0
t=xt. Since the input should not be omitted, we
setz0
t= 1for allt. Also, we do not use the boundary detector for the last layer. The hard sigm is
deﬁned by hard sigm (x) = max
0;min


deﬁned by hard sigm (x) = max
0;min
1;ax+1
2withabeing the slope variable.
Unlike the standard LSTM, the HM-LSTM has a top-down connection from (`+ 1) to`, which is
allowed to be activated only if a boundary is detected at the previous time step of the layer `(see
Eq. 6). This makes the layer `to be initialized with more long-term information after the boundary
is detected and execute the FLUSH operation. In addition, the input from the lower layer ( `1)
becomes effective only when a boundary is detected at the current time step in the layer ( `1) due
to the binary gate z`1
t. Figure 2 (left) shows the gating mechanism of the HM-LSTM at time step t.
Finally, the binary boundary state z`
tis obtained by:
z`
t=fbound(~z`
t): (8)
For the binarization function fbound :R!f0;1g, we can either use a deterministic step function:
z`
t=1if~z`
t>0:5
0otherwise;(9)
or sample from a Bernoulli distribution z`
tBernoulli (~z`


t>0:5
0otherwise;(9)
or sample from a Bernoulli distribution z`
tBernoulli (~z`
t). Although this binary decision is a key to
our model, it is usually difﬁcult to use stochastic gradient descent to train such model with discrete
decisions as it is not differentiable.
3.3 C OMPUTING GRADIENT OF BOUNDARY DETECTOR
Training neural networks with discrete variables requires more efforts since the standard backpropagation is no longer applicable due to the non-differentiability. Among a few methods for training a
neural network with discrete variables such as the REINFORCE (Williams, 1992; Mnih & Gregor,
2014) and the straight-through estimator (Hinton, 2012; Bengio et al., 2013), we use the straightthrough estimator to train our model. The straight-through estimator is a biased estimator because the
non-differentiable function used in the forward pass (i.e., the step function in our case) is replaced by
a differentiable function during the backward pass (i.e., the hard sigmoid function in our case). The
straight-through estimator, however, is much simpler and often works more efﬁciently in practice


straight-through estimator, however, is much simpler and often works more efﬁciently in practice
than other unbiased but high-variance estimators such as the REINFORCE. The straight-through
estimator has also been used in Courbariaux et al. (2016) and Vezhnevets et al. (2016).
The Slope Annealing Trick . In our experiment, we use the slope annealing trick to reduce the bias
of the straight-through estimator. The idea is to reduce the discrepancy between the two functions
used during the forward pass and the backward pass. That is, by gradually increasing the slope aof
the hard sigmoid function, we make the hard sigmoid be close to the step function. Note that starting
with a high slope value from the beginning can make the training difﬁcult while it is more applicable
later when the model parameters become more stable. In our experiments, starting from slope a= 1,
we slowly increase the slope until it reaches a threshold with an appropriate scheduling.
4 E XPERIMENTS
We evaluate the proposed model on two tasks, character-level language modelling and handwriting
sequence generation. Character-level language modelling is a representative example of discrete
6
Published as a conference paper at ICLR 2017
Penn Treebank
Model BPC


sequence generation. Character-level language modelling is a representative example of discrete
6
Published as a conference paper at ICLR 2017
Penn Treebank
Model BPC
Norm-stabilized RNN (Krueger & Memisevic, 2015) 1.48
CW-RNN (Koutník et al., 2014) 1.46
HF-MRNN (Mikolov et al., 2012) 1.41
MI-RNN (Wu et al., 2016) 1.39
MEn-gram (Mikolov et al., 2012) 1.37
BatchNorm LSTM (Cooijmans et al., 2016) 1.32
Zoneout RNN (Krueger et al., 2016) 1.27
HyperNetworks (Ha et al., 2016) 1.27
LayerNorm HyperNetworks (Ha et al., 2016) 1.23
LayerNorm CW-RNNy1.40
LayerNorm LSTMy1.29
LayerNorm HM-LSTM Sampling 1.27
LayerNorm HM-LSTM Soft1.27
LayerNorm HM-LSTM Step Fn. 1.25
LayerNorm HM-LSTM Step Fn. & Slope Annealing 1.24Hutter Prize Wikipedia
Model BPC
Stacked LSTM (Graves, 2013) 1.67


Model BPC
Stacked LSTM (Graves, 2013) 1.67
MRNN (Sutskever et al., 2011) 1.60
GF-LSTM (Chung et al., 2015) 1.58
Grid-LSTM (Kalchbrenner et al., 2015) 1.47
MI-LSTM (Wu et al., 2016) 1.44
Recurrent Memory Array Structures (Rocki, 2016a) 1.40
SF-LSTM (Rocki, 2016b)z1.37
HyperNetworks (Ha et al., 2016) 1.35
LayerNorm HyperNetworks (Ha et al., 2016) 1.34
Recurrent Highway Networks (Zilly et al., 2016) 1.32
LayerNorm LSTMy1.39
HM-LSTM 1.34
LayerNorm HM-LSTM 1.32
PAQ8hp12 (Mahoney, 2005) 1.32
decomp8 (Mahoney, 2009) 1.28
Table 1: BPC on the Penn Treebank test set (left) and Hutter Prize Wikipedia test set (right). ( ) This
model is a variant of the HM-LSTM that does not discretize the boundary detector states. ( y) These
models are implemented by the authors to evaluate the performance using layer normalization (Ba


models are implemented by the authors to evaluate the performance using layer normalization (Ba
et al., 2016) with the additional output module. ( z) This method uses test error signals for predicting
the next characters, which makes it not comparable to other methods that do not.
sequence modelling, where the discrete symbols form a distinct hierarchical multiscale structure. The
performance on real-valued sequences is tested on the handwriting sequence generation in which a
relatively clear hierarchical multiscale structure exists compared to other data such as speech signals.
4.1 C HARACTER -LEVEL LANGUAGE MODELLING
A sequence modelling task aims at learning the probability distribution over sequences by minimizing
the negative log-likelihood of the training sequences:
min
1
NNX
n=1TnX
t=1logp(xn
tjxn
<t;); (10)
whereis the model parameter, Nis the number of training sequences, and Tnis the length of the
n-th sequence. A symbol at time tof sequence nis denoted by xn
t, andxn
<tdenotes all previous
symbols at time t. We evaluate our model on three benchmark text corpora: (1) Penn Treebank, (2)


<tdenotes all previous
symbols at time t. We evaluate our model on three benchmark text corpora: (1) Penn Treebank, (2)
Text8 and (3) Hutter Prize Wikipedia. We use the bits-per-character (BPC), E[log2p(xt+1jxt)],
as the evaluation metric.
Model We use a model consisting of an input embedding layer, an RNN module and an output
module. The input embedding layer maps each input symbol into 128-dimensional continuous
vector without using any non-linearity. The RNN module is the HM-LSTM, described in Section 3,
with three layers. The output module is a feedforward neural network with two layers, an output
embedding layer and a softmax layer. Figure 2 (right) shows a diagram of the output module. At each
time step, the output embedding layer receives the hidden states of the three RNN layers as input. In
order to adaptively control the importance of each layer at each time step, we also introduce three
scalar gating units g`
t2Rto each of the layer outputs:
g`
t=sigm (w`[h1
t;;hL
t]); (11)
where w`2RPL


g`
t=sigm (w`[h1
t;;hL
t]); (11)
where w`2RPL
`=1dim (h`)is the weight parameter. The output embedding he
tis computed by:
he
t=ReLU LX
`=1g`
tWe
`h`
t!
; (12)
whereL= 3andReLU (x) = max(0;x)(Nair & Hinton, 2010). Finally, the probability distribution
for the next target character is computed by the softmax function, softmax (xj) =exjPK
k=1exk, where
each output class is a character.
7
Published as a conference paper at ICLR 2017
Text8
Model BPC
td-LSTM (Zhang et al., 2016) 1.63
HF-MRNN (Mikolov et al., 2012) 1.54
MI-RNN (Wu et al., 2016) 1.52
Skipping-RNN (Pachitariu & Sahani, 2013) 1.48
MI-LSTM (Wu et al., 2016) 1.44
BatchNorm LSTM (Cooijmans et al., 2016) 1.36
HM-LSTM 1.32


BatchNorm LSTM (Cooijmans et al., 2016) 1.36
HM-LSTM 1.32
LayerNorm HM-LSTM 1.29
Table 2: BPC on the Text8 test set.
Penn Treebank We process the Penn Treebank dataset (Marcus et al., 1993) by following the
procedure introduced in Mikolov et al. (2012). Each update is done by using a mini-batch of 64
examples of length 100to prevent the memory overﬂow problem when unfolding the RNN in time
for backpropagation. The last hidden state of a sequence is used to initialize the hidden state of the
next sequence to approximate the full backpropagation. We train the model using Adam (Kingma &
Ba, 2014) with an initial learning rate of 0:002. We divide the learning rate by a factor of 50when
the validation negative log-likelihood stopped decreasing. The norm of the gradient is clipped with a
threshold of 1(Mikolov et al., 2010; Pascanu et al., 2012). We also apply layer normalization (Ba
et al., 2016) to our models. For all of the character-level language modelling experiments, we apply
the same procedure, but only change the number of hidden units, mini-batch size and the initial
learning rate.


the same procedure, but only change the number of hidden units, mini-batch size and the initial
learning rate.
For the Penn Treebank dataset, we use 512units in each layer of the HM-LSTM and for the output
embedding layer. In Table 1 (left), we compare the test BPCs of four variants of our model to other
baseline models. Note that the HM-LSTM using the step function for the hard boundary decision
outperforms the others using either sampling orsoftboundary decision (i.e., hard sigmoid). The test
BPC is further improved with the slope annealing trick, which reduces the bias of the straight-through
estimator. We increased the slope awith the following schedule a= min (5 ;1 + 0:04Nepoch ), where
Nepoch is the maximum number of epochs. The HM-LSTM achieves test BPC score of 1:24. For the
remaining tasks, we ﬁxed the hard boundary decision using the step function without slope annealing
due to the difﬁculty of ﬁnding a good annealing schedule on large-scale datasets.
Text8 The Text8 dataset (Mahoney, 2009) consists of 100M characters extracted from the Wikipedia


Text8 The Text8 dataset (Mahoney, 2009) consists of 100M characters extracted from the Wikipedia
corpus. Text8 contains only alphabets and spaces, and thus we have total 27 symbols. In order to
compare with other previous works, we follow the data splits used in Mikolov et al. (2012). We use
1024 units for each HM-LSTM layer and 2048 units for the output embedding layer. The mini-batch
size and the initial learning rate are set to 128and0:001, respectively. The results are shown in
Table 2. The HM-LSTM obtains the state-of-the-art test BPC 1:29.
Hutter Prize Wikipedia The Hutter Prize Wikipedia ( enwik8 ) dataset (Hutter, 2012) contains
205symbols including XML markups and special characters. We follow the data splits used in Graves
(2013) where the ﬁrst 90M characters are used to train the model, the next 5M characters for validation,
and the remainders for the test set. We use the same model size, mini-batch size and the initial
learning rate as in the Text8. In Table 1 (right), we show the HM-LSTM achieving the test BPC 1:32,


learning rate as in the Text8. In Table 1 (right), we show the HM-LSTM achieving the test BPC 1:32,
which is a tie with the state-of-the-art result among the neural models. Although the neural models,
show remarkable performances, their compression performance is still behind the best models such
as PAQ8hp12 (Mahoney, 2005) and decomp8 (Mahoney, 2009).
Visualizing Learned Hierarchical Multiscale Structure In Figure 3 and 4, we visualize the
boundaries detected by the boundary detectors of the HM-LSTM while reading a character sequence
of total length 270taken from the validation set of either the Penn Treebank or Hutter Prize Wikipedia
dataset. Due to the page width limit, the ﬁgure contains the sequence partitioned into three segments
of length 90. The white blocks indicate boundaries z`
t= 1 while the black blocks indicate the
non-boundaries z`
t= 0.
Interestingly in both ﬁgures, we can observe that the boundary detector of the ﬁrst layer, z1, tends
to be turned on when it sees a space or after it sees a space, which is a reasonable breakpoint to
separate between words. This is somewhat surprising because the model self-organizes this structure
8


separate between words. This is somewhat surprising because the model self-organizes this structure
8
Published as a conference paper at ICLR 2017
Figure 3: Hierarchical multiscale structure in the Wikipedia dataset captured by the boundary
detectors of the HM-LSTM.
Penn Treebank Line 1Penn Treebank Line 2Penn Treebank Line 3
Figure 4: The `2-norm of the hidden states shown together with the states of the boundary detectors
of the HM-LSTM.
without any explicit boundary information. In Figure 3, we observe that the z1tends to detect the
boundaries of the words but also ﬁres within the words, where the z2tends to ﬁre when it sees either
an end of a word or 2;3-grams. In Figure 4, we also see ﬂushing in the middle of a word, e.g.,
“tele-FLUSH-phone”. Note that “tele” is a preﬁx after which a various number of postﬁxes can follow.
From these, it seems that the model uses to some extent the concept of surprise to learn the boundary.
Although interpretation of the second layer boundaries is not as apparent as the ﬁrst layer boundaries,


Although interpretation of the second layer boundaries is not as apparent as the ﬁrst layer boundaries,
it seems to segment at reasonable semantic / syntactic boundaries, e.g., “consumers may” - “want to
move their telephones a” - “little closer to the tv set <unk>”, and so on.
Another remarkable point is the fact that we do not pose any constraint on the number of boundaries
that the model can ﬁre up. The model, however, learns that it is more beneﬁcial to delay the
information ejection to some extent. This is somewhat counterintuitive because it might look more
beneﬁcial to feed the fresh update to the upper layers at every time step without any delay. We
conjecture the reason that the model works in this way is due to the FLUSH operation that poses an
implicit constraint on the frequency of boundary detection, because it contains both a reward (feeding
fresh information to upper layers) and a penalty (erasing accumulated information). The model ﬁnds
an optimal balance between the reward and the penalty.
To understand the update mechanism more intuitively, in Figure 4, we also depict the heatmap of the
`2-norm of the hidden states along with the states of the boundary detectors. As we expect, we can


`2-norm of the hidden states along with the states of the boundary detectors. As we expect, we can
see that there is no change in the norm value within segments due to the COPY operation. Also, the
color ofkh1kchanges quickly (at every time step) because there is no COPY operation in the ﬁrst
layer. The color of kh2kchanges less frequently based on the states of z1
tandz2
t1. The color of
kh3kchanges even slowly, i.e., only when z2
t= 1.
A notable advantage of the proposed architecture is that the internal process of the RNN becomes
more interpretable. For example, we can substitute the states of z1
tandz2
t1into Eq. 2 and infer
which operation among the UPDATE, COPY and FLUSH was applied to the second layer at time step
t. We can also inspect the update frequencies of the layers simply by counting how many UPDATE
and FLUSH operations were made in each layer. For example in Figure 4, we see that the ﬁrst layer
updates at every time step (which is 270UPDATE operations), the second layer updates 56times,
9
Published as a conference paper at ICLR 2017
IAM-OnDB
Model Average Log-Likelihood
Standard LSTM 1081


9
Published as a conference paper at ICLR 2017
IAM-OnDB
Model Average Log-Likelihood
Standard LSTM 1081
HM-LSTM 1137
HM-LSTM & Slope Annealing 1167
Table 3: Average log-likelihood per sequence on the IAM-OnDB test set.
the ground truth of pen-tip locationVisualization by segments usingVisualization by segments usingthe states ofz2
Figure 5: The visualization by segments based on either the given pen-tip location or states of the z2.
and only 9updates has made in the third layer. Note that, by design, the ﬁrst layer performs UPDATE
operation at every time step and then the number of UPDATE operations decreases as the layer level
increases. In this example, the total number of updates is 335for the HM-LSTM which is 60% of
reduction from the 810updates of the standard RNN architecture.
4.2 H ANDWRITING SEQUENCE GENERATION
We extend the evaluation of the HM-LSTM to a real-valued sequence modelling task using IAMOnDB (Liwicki & Bunke, 2005) dataset. The IAM-OnDB dataset consists of 12;179handwriting


examples, each of which is a sequence of (x;y)coordinate and a binary indicator pfor pen-tip
location, giving us (x1:Tn;y1:Tn;p1:Tn), wherenis an index of a sequence. At each time step,
the model receives (xt;yt;pt), and the goal is to predict (xt+1;yt+1;pt+1). The pen-up ( pt= 1)
indicates an end of a stroke, and the pen-down ( pt= 0) indicates that a stroke is in progress. There
is usually a large shift in the (x;y)coordinate to start a new stroke after the pen-up happens. We
remove all sequences whose length is shorter than 300. This leaves us 10;465sequences for training,
581for validation, 582for test. The average length of the sequences is 648. We normalize the range
of the (x;y)coordinates separately with the mean and standard deviation obtained from the training
set. We use the mini-batch size of 32, and the initial learning rate is set to 0:0003 .
We use the same model architecture as used in the character-level language model, except that the
output layer is modiﬁed to predict real-valued outputs. We use the mixture density network as the


output layer is modiﬁed to predict real-valued outputs. We use the mixture density network as the
output layer following Graves (2013), and use 400units for each HM-LSTM layer and for the output
embedding layer. In Table 3, we compare the log-likelihood averaged over the test sequences of the
IAM-OnDB dataset. We observe that the HM-LSTM outperforms the standard LSTM. The slope
annealing trick further improves the test log-likelihood of the HM-LSTM into 1167 in our setting. In
this experiment, we increased the slope awith the following schedule a= min (3 ;1 + 0:004Nepoch ).
In Figure 5, we let the HM-LSTM to read a randomly picked validation sequence and present the
visualization of handwriting examples by segments based on either the states of z2or the states of
pen-tip location3.
5 C ONCLUSION
In this paper, we proposed the HM-RNN that can capture the latent hierarchical structure of the
sequences. We introduced three types of operations to the RNN, which are the COPY , UPDATE
and FLUSH operations. In order to implement these operations, we introduced a set of binary
variables and a novel update rule that is dependent on the states of these binary variables. Each binary


and FLUSH operations. In order to implement these operations, we introduced a set of binary
variables and a novel update rule that is dependent on the states of these binary variables. Each binary
variable is learned to ﬁnd segments at its level, therefore, we call this binary variable, a boundary
detector. On the character-level language modelling, the HM-LSTM achieved state-of-the-art result
on the Text8 dataset and comparable results to the state-of-the-art results on the Penn Treebank
and Hutter Prize Wikipedia datasets. Also, the HM-LSTM outperformed the standard LSTM on
the handwriting sequence generation. Our results and analysis suggest that the proposed HM-RNN
can discover the latent hierarchical structure of the sequences and can learn efﬁcient hierarchical
multiscale representation that leads to better generalization performance.
3The plot function could be found at blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/ .
10
Published as a conference paper at ICLR 2017
ACKNOWLEDGMENTS
The authors would like to thank Alex Graves, Tom Schaul and Hado van Hasselt for their fruitful
comments and discussion. We acknowledge the support of the following agencies for research


The authors would like to thank Alex Graves, Tom Schaul and Hado van Hasselt for their fruitful
comments and discussion. We acknowledge the support of the following agencies for research
funding and computing support: Ubisoft, Samsung, IBM, Facebook, Google, Microsoft, NSERC,
Calcul Québec, Compute Canada, the Canada Research Chairs and CIFAR. The authors thank the
developers of Theano (Team et al., 2016). JC would like to thank Arnaud Bergenon and Frédéric
Bastien for their technical support. JC would also like to thank Guillaume Alain, Kyle Kastner and
David Ha for providing us useful pieces of code.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 ,
2016.
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Yoshua Bengio, et al. End-to-end attention-based large
vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP) , pp. 4945–4949. IEEE, 2016.


vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP) , pp. 4945–4949. IEEE, 2016.
Yoshua Bengio. Learning deep architectures for ai. Foundations and trends R
in Machine Learning , 2(1):1–127,
2009.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic
neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large
vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech
and Signal Processing (ICASSP) , pp. 4960–4964. IEEE, 2016.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua
Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In
Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) , October 2014.


Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) , October 2014.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural
networks. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , 2015.
Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. A character-level decoder without explicit segmentation
for neural machine translation. Association for Computational Linguistics (ACL) , 2016.
Tim Cooijmans, Nicolas Ballas, César Laurent, and Aaron Courville. Recurrent batch normalization. arXiv
preprint arXiv:1603.09025 , 2016.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural
networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint
arXiv:1602.02830 , 2016.
Salah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term dependencies. In
Advances in Neural Information Processing Systems , pp. 493–499. Citeseer, 1995.


Advances in Neural Information Processing Systems , pp. 493–499. Citeseer, 1995.
Santiago Fernández, Alex Graves, and Jürgen Schmidhuber. Sequence labelling in structured domains with
hierarchical recurrent neural networks. In Proceedings of the 20th international joint conference on Artiﬁcal
intelligence , pp. 774–779. Morgan Kaufmann Publishers Inc., 2007.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016.
G. Hinton. Neural networks for machine learning. Coursera, video lectures, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735–1780,
1997.
Judy Hoffman, Eric Tzeng, Jeff Donahue, Yangqing Jia, Kate Saenko, and Trevor Darrell. One-shot adaptation
of supervised deep convolutional models. arXiv preprint arXiv:1312.6204 , 2013.


of supervised deep convolutional models. arXiv preprint arXiv:1312.6204 , 2013.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth.
arXiv preprint arXiv:1603.09382 , 2016.
11
Published as a conference paper at ICLR 2017
Marcus Hutter. The human knowledge compression contest. 2012. URL http://prize.hutter1.net/ .
Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. arXiv preprint
arXiv:1507.01526 , 2015.
Yoon Kim. Convolutional neural networks for sentence classiﬁcation. arXiv preprint arXiv:1408.5882 , 2014.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models.
arXiv preprint arXiv:1508.06615 , 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
2014.


Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.
Lingpeng Kong, Chris Dyer, and Noah A Smith. Segmental recurrent neural networks. arXiv preprint
arXiv:1511.06018 , 2015.
Jan Koutník, Klaus Greff, Faustino Gomez, and Jürgen Schmidhuber. A clockwork rnn. In Proceedings of the
31st International Conference on Machine Learning (ICML 2014) , 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classiﬁcation with deep convolutional neural
networks. In Advances in Neural Information Processing Systems , pp. 1097–1105, 2012.
David Krueger and Roland Memisevic. Regularizing rnns by stabilizing activations. arXiv preprint
arXiv:1511.08400 , 2015.


David Krueger and Roland Memisevic. Regularizing rnns by stabilizing activations. arXiv preprint
arXiv:1511.08400 , 2015.
David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh
Goyal, Yoshua Bengio, Hugo Larochelle, Aaron Courville, et al. Zoneout: Regularizing rnns by randomly
preserving hidden activations. arXiv preprint arXiv:1606.01305 , 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436–444, 2015.
Tsungnan Lin, Bill G Horne, Peter Tino, and C Lee Giles. Learning long-term dependencies in narx recurrent
neural networks. IEEE Transactions on Neural Networks , 7(6):1329–1338, 1996.
Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. Character-based neural machine translation. arXiv
preprint arXiv:1511.04586 , 2015.
Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten


Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten
text on a whiteboard. In Eighth International Conference on Document Analysis and Recognition (ICDAR’05) ,
pp. 956–961. IEEE, 2005.
Matthew V Mahoney. Adaptive weighing of context models for lossless data compression. 2005.
Matthew V Mahoney. Large text compression benchmark. URL: http://www. mattmahoney. net/text/text. html ,
2009.
Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of
english: The penn treebank. Computational linguistics , 19(2):313–330, 1993.
Tomas Mikolov, Martin Karaﬁát, Lukas Burget, Jan Cernock `y, and Sanjeev Khudanpur. Recurrent neural
network based language model. In INTERSPEECH , volume 2, pp. 3, 2010.
Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and J Cernocky. Subword language modeling with neural networks. Preprint , 2012. URL http://www.fit.vutbr.cz/


~imikolov/rnnlm/char.pdf .
Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Proceedings of
the 31st International Conference on Machine Learning (ICML-14) , pp. 1791–1799, 2014.
Michael C Mozer. Induction of multiscale temporal structure. Advances in neural information processing
systems , pp. 275–275, 1993.
Vinod Nair and Geoffrey E Hinton. Rectiﬁed linear units improve restricted boltzmann machines. In Proceedings
of the 27th International Conference on Machine Learning (ICML-10) , pp. 807–814, 2010.
Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are
they needed? arXiv preprint arXiv:1301.5650 , 2013.
12
Published as a conference paper at ICLR 2017
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difﬁculty of training recurrent neural networks.
arXiv preprint arXiv:1211.5063 , 2012.
Kamil M Rocki. Recurrent memory array structures. arXiv preprint arXiv:1607.03085 , 2016a.


Kamil M Rocki. Recurrent memory array structures. arXiv preprint arXiv:1607.03085 , 2016a.
Kamil M Rocki. Surprisal-driven feedback in recurrent networks. arXiv preprint arXiv:1608.06027 , 2016b.
Jürgen Schmidhuber. Neural sequence chunkers. 1991.
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural
Computation , 4(2):234–242, 1992.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks , 61:85–117, 2015.
Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie.
A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of
the 24th ACM International on Conference on Information and Knowledge Management , pp. 553–562. ACM,
2015.
Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout:


Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout:
a simple way to prevent neural networks from overﬁtting. Journal of Machine Learning Research , 15(1):
1929–1958, 2014.
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In
Proceedings of the 28th International Conference on Machine Learning (ICML’11) , pp. 1017–1024, 2011.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances
in Neural Information Processing Systems , pp. 3104–3112, 2014.
The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller,
Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, et al. Theano: A python
framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 , 2016.


framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688 , 2016.
Alexander Vezhnevets, V olodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves, Oriol Vinyals, Koray
Kavukcuoglu, et al. Strategic attentive writer for learning macro-actions. arXiv preprint arXiv:1606.04695 ,
2016.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption
generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 3156–
3164, 2015.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.
Machine learning , 8(3-4):229–256, 1992.
Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan Salakhutdinov. On multiplicative
integration with recurrent neural networks. arXiv preprint arXiv:1606.06630 , 2016.
Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, and Yoshua


Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, and Yoshua
Bengio. Architectural complexity measures of recurrent neural networks. arXiv preprint arXiv:1602.08210 ,
2016.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway
networks. arXiv preprint arXiv:1607.03474 , 2016.
13


ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 1
Bottom-up Instance Segmentation using
Deep Higher-Order CRFs
Anurag Arnab
anurag.arnab@eng.ox.ac.uk
Philip H.S. Torr
philip.torr@eng.ox.ac.ukDepartment of Engineering Science
University of Oxford
United Kingdom
Abstract
Traditional Scene Understanding problems such as Object Detection and Semantic
Segmentation have made breakthroughs in recent years due to the adoption of deep learning. However, the former task is not able to localise objects at a pixel level, and the latter
task has no notion of different instances of objects of the same class. We focus on the task
of Instance Segmentation which recognises and localises objects down to a pixel level.
Our model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable higher order potentials based on object detector outputs. This allows us to reason about instances
from an initial, category-level semantic segmentation. Our simple method effectively
leverages the great progress recently made in semantic segmentation and object detection. The accurate instance-level segmentations that our network produces is reﬂected by
the considerable improvements obtained over previous work at high APrIoU thresholds.

Summary:
Traditional Scene Understanding problems such as Object Detection and
Semantic Segmentation have made breakthroughs in recent years due to the
adoption of deep learning. However, the former task is not able to localise
objects at a pixel level, and the latter task has no notion of different
instances of objects of the same class. We focus on the task of Instance
Segmentation which recognises and localises objects down to a pixel level. Our
model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable
higher order potentials based on object detector outputs. This allows us to
reason about instances from an initial, category-level semantic segmentation.
Our simple method effectively leverages the great progress recently made in
semantic segmentation and object detection. The accurate instance-level
segmentations that our network produces is reflected by the considerable
improvements obtained over previous work.

British Machine Vision Conference (BMVC) 2016


the considerable improvements obtained over previous work at high APrIoU thresholds.
1 Introduction
Object detection and semantic segmentation have been two of the most popular Scene Understanding problems within the Computer Vision community. Great advances have been made
in recent years due to the adoption of deep learning and Convolutional Neural Networks
[19, 22, 32]. In this paper, we focus on the problem of Instance Segmentation . Instance
Segmentation lies at the intersection of Object Detection – which localises different objects
at a bounding box level, but does not segment them – and Semantic Segmentation – which
determines the object-class label of each pixel in the image, but has no notion of different
instances of the same class. As shown in Figure 1, the task of instance segmentation localises
objects to a pixel level.
Many recent instance segmentation works have built on the “Simultaneous Detection
and Segmentation” (SDS) approach of Hariharan et al. [13]. These methods [4, 5, 13, 14]
generate object proposals [1], classify each proposal into an object category, and then reﬁne
the bounding box proposal into a segmentation of the primary object the proposal contains.
However, because these methods localise an object before segmenting it, they are restricted


the bounding box proposal into a segmentation of the primary object the proposal contains.
However, because these methods localise an object before segmenting it, they are restricted
by the quality of the initial proposal. Moreover, the proposal generator [1] these works used
takes about 40 seconds to process an image.
c/circlecopyrt2016. The copyright of this document resides with its authors.
It may be distributed unchanged freely in print or electronic forms.
arXiv:1609.02583v1  [cs.CV]  8 Sep 2016
2 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation
Figure 1: Our method uses the results of an object detector and the original image (a) to produce a category-level semantic segmentation (b). The semantic segmentation result, object
detection bounding boxes, and recalibrated detection scores (§3.2) are then used to reﬁne
the semantic segmentation into an instance segmentation (c). The semantic segmentation,
in which a large number of pixels have been correctly labelled as “person”, does not tell us


the semantic segmentation into an instance segmentation (c). The semantic segmentation,
in which a large number of pixels have been correctly labelled as “person”, does not tell us
how many people there are and where each person is in the image. The instance segmentation answers these questions. Also note how our method is not affected by the false-positive
“bottle” detections.
We present a different approach to instance segmentation, where we initially perform
a category-level, semantic segmentation of the image before reasoning about the different
instances in the scene. We can identify instances because our semantic segmentation network
incorporates a Higher Order Conditional Random Field (CRF) which uses cues from the
output of an object detector. This CRF is fully differentiable and can be inserted as a layer
of a neural network [2, 37]. As a result, it can be used as a simple extension to an existing
semantic segmentation network to perform the related task of instance segmentation.
Our simple, bottom-up method is able to effectively leverage the progress made by stateof-the-art semantic segmentation and object detection networks to perform the related task
of instance segmentation. We show this by evaluating our approach on the PASCAL VOC
2012 [8] dataset, using the standard APrmeasure [13]. The fact that our system is able to


2012 [8] dataset, using the standard APrmeasure [13]. The fact that our system is able to
produce accurate instance segmentations is reﬂected by the considerable improvements that
we make over previous work at high overlap thresholds.
2 Related Work
Instance segmentation is closely related to the widely-studied problems of semantic segmentation and object detection. We thus review these two tasks ﬁrst.
Semantic Segmentation: This task was traditionally performed by extracting dense features from an image and classifying them on a per-pixel or per-region basis [16, 31]. The
individual predictions made by these classiﬁers were often noisy as they lacked global context. As a result, they were post-processed with a CRF whose priors encourages nearby
pixels, and pixels of similar appearance, to share the same class label. The CRF of [31]
initially contained only unary terms (obtained from the classiﬁer) and 8-connected pairwise
terms. Subsequent improvements included introducing densely-connected pairwise potentials [18], using higher order potentials encouraging consistency over cliques larger than two
pixels [17, 35], modelling the co-occurrence of labels [20, 35] and utilising the results of
object detectors [21, 33, 36].


pixels [17, 35], modelling the co-occurrence of labels [20, 35] and utilising the results of
object detectors [21, 33, 36].
Using CNNs for semantic segmentation can greatly improve the unary, per-pixel predictions, as shown by [27]. Chen et al. showed further improvements by post-processing the
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 3
CNN output with a CRF [3]. Current state-of-the-art methods [2, 25, 37] incorporate inference of a CRF as layers within a deep network which perform the differentiable mean ﬁeld
inference algorithm. In our work, we extend the end-to-end trainable, higher-order, object
detection potentials proposed by [2] to the task of instance segmentation. Additionally, unlike the aforementioned works which always assume a ﬁxed number of labels in the CRF at
all times, our system includes a CRF where the number of labels varies per input image.
Object Detection: Part-based models [9] were popular before CNNs, which had been
shown to excel in the task of image classiﬁcation [19, 32], were adapted to object detection. Recent detection algorithms are based on using CNNs to classify object proposals


shown to excel in the task of image classiﬁcation [19, 32], were adapted to object detection. Recent detection algorithms are based on using CNNs to classify object proposals
and then performing bounding box regression to reﬁne the initial proposals’ bounding boxes
[10, 11, 12, 28]. This pipeline can be accelerated by computing convolutional features over
the whole image, and then pooling features for each proposal [11, 15]. Object proposals
were initially hand-crafted [1, 34], but this step has now been integrated into CNNs [28].
Instance Segmentation: This task was popularised by Hariharan et al. [13]. Their approach was similar to object detection pipelines in that region proposals were ﬁrst generated and classiﬁed into different object categories before using bounding box regression as
post-processing. A class-speciﬁc segmentation was then performed to simultaneously detect
and segment an object. However, approaches that segment instances by reﬁning detections
[4, 5, 13, 14] are inherently limited by the quality of the initial proposals, and the systems
cannot recover from errors made by the initial detector. Furthermore, since these methods
generate and classify multiple overlapping region proposals per image, they cannot actually


cannot recover from errors made by the initial detector. Furthermore, since these methods
generate and classify multiple overlapping region proposals per image, they cannot actually
produce segmentation maps of the form shown in Figures 1(b) and (c) [24]. Liang et al. [23]
address some of these shortcomings by iteratively reﬁning initial object proposals, whilst [7]
generates box-proposals, creates masks from these proposals and then classiﬁes these masks
in a single network trained with three loss functions summed together.
A proposal-free method was recently presented by [24] where a semantic segmentation of the image is ﬁrst performed using the network of [3]. Thereafter the category-level
segmentation output, along with CNN features, are used to predict instance-level bounding
boxes for the objects in the image. The number of instances of each object category are also
predicted in order to facilitate the ﬁnal clustering step. An alternate method was proposed
by [29] where a Recurrent Neural Network outputs an object instance at each time step. This
method, however, has not been evaluated on multiple classes.


by [29] where a Recurrent Neural Network outputs an object instance at each time step. This
method, however, has not been evaluated on multiple classes.
Our method also ﬁrst performs a bottom-up, semantic segmentation of the image. However, our semantic segmentation network uses the outputs of an object detector as an additional cue in its ﬁnal CRF layer. During inference, each object hypothesis from the detector
is evaluated in light of other energies in the CRF. Thus, the relative score of false positive
detections can be decreased, and correct detections increased. These rescored detections
are then used to identify instances from the initial semantic segmentation. The fact that our
system combines information from semantic segmentation and object detection networks allows us to bypass the complexity of the “instance bounding box” and “instance number”
prediction steps used by [24].
3 Proposed Approach
Our proposed method ﬁrst performs a semantic segmentation of the input image, classifying
each pixel into one of K+1 categories where Kis the number of foreground classes. The
resulting semantic segmentation is then reﬁned into an instance-level segmentation, where
4 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Pixelwise CNN  Higher Order CRF


4 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Pixelwise CNN  Higher Order CRF
Instance Identification  Instance CRF
K+1
Object Detector
D+1  End-to-end object segmentation subnetwork
End-to-end Instance segmentation subnetwork
Input
Output  Object
Segmentation
Identified
Instances
Figure 2: Overview of our end-to-end method. Our system ﬁrst computes a category-level
semantic segmentation of the image. A CRF with higher-order detection potentials is used
to obtain this result in a semantic segmentation subnetwork. This results in a W×H×(K+
1)dimensional volume where Kis the number of foreground classes. WandHare the
image’s width and height respectively. The original detections and the detector conﬁdences
recalibrated by the Higher Order CRF are used to identify instances in the image, producing
aW×H×(D+1)dimensional volume where Dis the number of detections (variable per
image). Thereafter, another Instance CRF is used to compute the ﬁnal result.


image). Thereafter, another Instance CRF is used to compute the ﬁnal result.
the object class of each instance segment is obtained from the previous semantic segmentation. Both of these stages, while conceptually different, are fully differentiable and the entire
system can be implemented as a neural network.
Figure 2 shows our category-level segmentation network, consisting of a pixelwise CNN
[27] followed by a fully differentiable CRF with object detection potentials, as described in
Section 3.2. The object detection results that our CRF takes in, as well as the recalibrated
detection scores that it outputs, allows us to identify object instances, and obtain a ﬁnal
instance-level segmentation in the next stage of our pipeline. This is described in Section
3.3. However, we ﬁrst review Conditional Random Fields and introduce the notation used in
this paper.
3.1 Conditional Random Fields
Assume an image Iwith Npixels, indexed 1 ,2...N, and deﬁne a set of random variables,
X1,X2,...,XN, one for every pixel. In a generic labelling problem, we assign every pixel a


X1,X2,...,XN, one for every pixel. In a generic labelling problem, we assign every pixel a
label from a predeﬁned set of labels Lsuch that each Xi∈L. Let X= [X1X2...XN]T. In
this case, any particular assignment xtoXis a solution to our labelling problem.
In the case of semantic segmentation, we assign each pixel, Xi, a label corresponding
to object categories such as “person” and “car”. In instance segmentation, the labels are
drawn from the product label space of object categories and instance numbers. Examples are
“person_1” and “person_2” as shown by the different colours in Figure 1.
Given a graph Gwhere the vertices are from {X}and connections between these variables are deﬁned by edges, the pair (I,X)is modelled as a CRF deﬁned by Pr (X=x|I) =
(1/Z(I))exp(−E(x|I)). The term, E(x|I)is the energy of the labelling xandZ(I)is the
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 5


ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 5
data-dependent partition function. From here onwards, we drop the conditioning on Ito
keep the notation uncluttered. The energy of an assignment is deﬁned over the set of cliques
Cin the graph Gas
E(x) =∑
c∈Cψc(xc), (1)
where xcis the vector formed by selecting elements of xthat correspond to random variables
belonging to the clique c, andψc(.)is the cost function for a clique c.
The energy function considered by many semantic segmentation works, E(x) =∑iψU
i(xi)+
∑i<jψP
i j(xi,xj), consists of only unary (cliques of size one) and densely-connected pairwise
potentials (cliques of size two) [3, 18, 37]. When dense pairwise potentials are used in a
CRF to obtain higher accuracy, exact inference is not tractable and approximate inference
methods such as mean ﬁeld are used [18].
3.2 Higher Order Detection Potentials
In addition to the standard unary and densely-connected pairwise terms, we also include a


3.2 Higher Order Detection Potentials
In addition to the standard unary and densely-connected pairwise terms, we also include a
higher order term based on object detections. Intuitively, a state-of-the-art object detector
[11, 28] can help segmentation performance in cases where the detector correctly predicts an
object at a location in which the pixels have been misclassiﬁed by the unary classiﬁer, and
thus reduce the recognition error [6]. Arnab et al. [2] formulated object detection potentials
in a manner that is amenable to the differentiable mean ﬁeld inference algorithm so that it
could be integrated into a neural network, and showed that it does indeed improve semantic
segmentation performance. We review the detection potentials described in [2] as they also
help us to reason about instances in an input image, a problem not considered by [2].
Assume that we have Dobject detections for an input image (this number varies for every
image), and that the dthdetection is of the form (ld,sd,Fd,Bd)where ld∈Lis the object class
label of the detected object, sdis the detector’s conﬁdence score, Fd⊂{1,2,...N}is the set


label of the detected object, sdis the detector’s conﬁdence score, Fd⊂{1,2,...N}is the set
of indices of the pixels belonging to the foreground of the detection and Bd⊂{1,2,...N}is
the set of indices falling in the detection’s bounding box. The foreground is obtained from
a foreground/background segmentation method such as GrabCut [30] and indicates a rough
segmentation of the detected object. The detection potentials should encourage the set of
pixels represented by Fdto take the object class label, ld. However, this should be a soft
constraint, since the foreground segmentation could be inaccurate and the entire detection
itself could be a false-positive. The detection dshould also be identiﬁed as invalid if other
energies in the CRF strongly suggest that many pixels in Fddo not take the label ld.
This is formulated by introducing a latent binary random variable, Y1,Y2...YDfor every
detection. If the dthdetection has been found to be valid after inference, Ydwill be set to 1,
and 0 otherwise. The ﬁnal value of Ydis determined probabilistically by mean ﬁeld inference.


and 0 otherwise. The ﬁnal value of Ydis determined probabilistically by mean ﬁeld inference.
All latent Ydvariables are added to the CRF which previously only contained Xivariables.
Let each (Xd,Yd), where{Xd}={Xi∈{X}|i∈Fd}, form a clique in the CRF. An assignment
(xd,yd)to the clique (Xd,Yd)has the energy:
ψDet
d(Xd=xd,Yd=yd) =

wlsd
|Fd|∑|Fd|
i=1[x(i)
d=ld]ifyd=0,
wlsd
|Fd|∑|Fd|
i=1[x(i)
d/negationslash=ld]ifyd=1,(2)
where x(i)
dis the ithelement in vector xd,[.]is the Iverson bracket and wlis a class-speciﬁc,
learnable weight parameter. This potential encourages consistency among X(i)
dvariables and
6 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION


learnable weight parameter. This potential encourages consistency among X(i)
dvariables and
6 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Ydsince it encourages X(i)
dvariables to take the label ldwhen Ydis 1, and also encourages Yd
to be 0 when many X(i)
dvariables do not take the label ld.
A unary potential for the latent Yvariables is also included, which is initialised to the
conﬁdence score sdof the object detector (in addition to the joint potential with the Xvariables). This initial conﬁdence then changes throughout CRF inference, depending on the
other potentials. The ﬁnal probability of the latent Yvariable can thus be viewed as a recalibrated detection score, which gets adjusted during inference based on how much it agrees
with the segmentation unaries and pairwise potentials. Note that this “recalibration property”
of the latent Yvariables was not considered by [2]. The ﬁnal form of the energy function
(Eq 1), is thus
E(x) =∑
iψU
i(xi)+∑
i<jψP
i,j(xi,xj)+∑
dψDet


iψU
i(xi)+∑
i<jψP
i,j(xi,xj)+∑
dψDet
d(xd,yd)+∑
dψU
d(yd). (3)
3.3 Instance Identiﬁcation and Reﬁnement
Once we have a category-level segmentation of the image, each pixel still needs to be assigned to an object instance. We assume that each object detection represents a possible
instance. Since there are Dobject detections (where Dvaries for every image), and some
pixels do not belong to any object instance, but are part of the background, we have a labelling problem involving D+1 labels. Our set of labels, D, is thus{1,2,...D+1}.
A naïve solution to the problem of recovering instances from a category-level segmentation would be as follows: If a pixel falls within the bounding box of a detection, and its
category label agrees with the class predicted by the object detector, then it is assigned to the
instance represented by that detection. However, this method cannot deal with overlapping
bounding boxes which are typical when objects occlude one another (as in Figs. 1 and 2).
Instead, if a pixel falls within the bounding box of a detection, we assign the pixel to


Instead, if a pixel falls within the bounding box of a detection, we assign the pixel to
that instance with a probability proportional to the rescored detection (obtained from the
probability of the latent Yvariable after inference) and the semantic segmentation conﬁdence
for that class, as shown in Equation 4
Pr(vi=k) =/braceleftBigg1
Z(Y,Q)Qi(lk)Pr(Yk=1)ifi∈Bk
0 otherwise .(4)
Here, viis a multinomial random variable indicating the “identiﬁed instance” at pixel iand
takes on labels from D,Qi(l)is the output of the initial category-level segmentation stage
of our network and denotes the probability of pixel itaking the label l∈L, and Z(Y,Q)
normalises the probability in case there are multiple bounding boxes overlapping the same
pixel. For this formulation to be valid, we also add another detection, d0, denoting background pixels that are not overlapped by any detection. This then acts as the unary potentials
of another CRF with the energy:
E(v) =∑
iψU
i(vi)+∑
i<jψP


of another CRF with the energy:
E(v) =∑
iψU
i(vi)+∑
i<jψP
i,j(vi,vj) ψU
i(vi) =−lnPr(vi). (5)
This Instance CRF contains unary, ψU
i(vi), and densely-connected pairwise terms, ψP
i,j(vi,vj),
encouraging appearance and spatial consistency [18]. These priors are valid in the case of
instance segmentation as well. We then perform mean ﬁeld inference on this ﬁnal Instance
CRF to get our ﬁnal output. Note that this ﬁnal, pairwise CRF is dynamic – the D+1 labels
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 7
that it deals with are different for every image, where Dis equal to the number of object detections obtained after non-maximal suppression. This differs with the CRFs considered in
semantic segmentation literature which always have a ﬁxed number of labels for all images


semantic segmentation literature which always have a ﬁxed number of labels for all images
in the dataset [17, 18, 35, 37]. Zheng et al. [37] showed how the iterative mean ﬁeld inference algorithm can be unrolled into a series of differentiable operations and be implemented
as a recurrent neural network. We employ a similar method so that our dynamic CRF is fully
differentiable and can thus be incorporated into a neural network. This is achieved by using
CRF weights that are not class-speciﬁc, and can thus generalise to any number of labels per
image. A label here refers to an instance associated with a detection and has no semantic
meaning. For example, in Fig. 1, label “1” was associated with a dining table, whilst in Fig.
2 it is a sheep. It therefore does not make sense to have class-speciﬁc weights in any case.
This approach to instance segmentation is bottom-up, in the sense that we ﬁrst perform
category-level segmentation of each pixel before predicting instances. However, the fact
that we used object detection cues in performing our semantic segmentation helps us to
subsequently reason about instances. Moreover, the fact that we used results from both
semantic segmentation and object detection makes sense since instance segmentation can be


subsequently reason about instances. Moreover, the fact that we used results from both
semantic segmentation and object detection makes sense since instance segmentation can be
regarded as being at the intersection of these two more common problems.
4 Experimental Results and Analysis
We ﬁrst describe the dataset and our experimental setup (§4.1), before examining the efﬁcacy
of our detection potentials (§4.2) and comparing our approach with other recent instance
segmentation methods (§4.3).
4.1 Experimental Setup
Following previous work, we evaluate our instance segmentation performance on the PASCAL VOC 2012 [8] validation set which comprises of 1449 images with high-quality annotations. There is no test server for instance segmentation to evaluate on the test set. We
use the standard APrmeasure for evaluation [13], which computes the mean average precision under different Intersection over Union (IoU) scores between the predicted and ground
truth segmentations (rather than IoU between bounding boxes, as in object detection). In object detection, a predicted bounding box is considered correct if the IoU between it and the
ground-truth bounding box is greater than 0.5. Here, we consider different overlap thresholds
between the predicted and ground-truth segments, since higher overlap thresholds require the


ground-truth bounding box is greater than 0.5. Here, we consider different overlap thresholds
between the predicted and ground-truth segments, since higher overlap thresholds require the
segmentations to be more precise and are therefore better indicators of segmentation performance. We also quote the APr
volmeasure, which is the mean APrscore for overlap thresholds
ranging from 0 .1 to 0 .9 in increments of 0 .1 [13]. The detailed annotations in the VOC 2012
dataset allows us to reliably evaluate the AProf our method at high overlap thresholds.
We ﬁrst trained a network for semantic segmentation, in a similar manner to [27] and
[37]. Although the trained models from these two authors have been made publicly available,
we could not use them since they had been trained on part of the VOC validation set. We
ﬁrst trained a fully convolutional network [27] (ﬁnetuned from VGG-16 trained on ImageNet
[32]) using VOC 2012 training data, augmented with images from the Semantic Boundaries
Dataset [13] which do not overlap with the VOC Validation set, and the Microsoft COCO
dataset [26].


Dataset [13] which do not overlap with the VOC Validation set, and the Microsoft COCO
dataset [26].
To this pretrained network, we added the Higher Order CRF, and ﬁnetuned with a learning rate of 10−11using only VOC 2012 data which is ﬁnely annotated. The learning rate is
8 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Table 1: Comparison of instance segmentation performance against baselines on the VOC
2012 Validation Set of 1449 images. We report the APrmeasure at different IoU thresholds.
Method APrat 0.5 APrat 0.6 APrat 0.7 APr
vol
Baseline without detection potentials 54.6 48.5 41.8 50.0
Baseline with detection potentials,
butYvariables ignored57.5 51.6 44.5 52.4
Full system with detection potentials,
and recalibrated detection scores from
Yvariables58.3 52.4 45.4 53.1
low since the loss was not normalised by the number of pixels in the training image. When
training our Higher Order CRF, we use the publicly available Faster-RCNN object detection
framework [28]. The semantic segmentation performance on the VOC validation set is a


training our Higher Order CRF, we use the publicly available Faster-RCNN object detection
framework [28]. The semantic segmentation performance on the VOC validation set is a
mean IoU of 73.4% when using only pairwise potentials in the CRF. This rises to 75.3%
when using detection potentials as well. The authors of [2] observed a similar increase when
using detection potentials.
4.2 Effect of detection potentials
To analyse the effect of our detection potentials, we evaluate our system by disabling the
higher order potentials in our Semantic Segmentation CRF. Because our detection potentials
are disabled, we can no longer use the output of the latent Yvariables as our recalibrated
detection score, and use the original detector’s conﬁdence score instead. As shown in Table
1, the result is that the APrat 0.5 is 3.7% lower and the APr
volis 3.1% lower.
The detection potentials improve instance segmentation performance as they improve
both our initial semantic segmentation, and also recalibrate the detector’s initial scores. To
decouple these two effects, we then include detection potentials in our Segmentation CRF,
but ignore the latent Yvariables output by the CRF and use the detector’s original conﬁdence


but ignore the latent Yvariables output by the CRF and use the detector’s original conﬁdence
score instead as the input to the instance segmentation network. The second row of Table
1 shows that this second baseline has an APrat 0.5 and an APr
volthat are respectively 0.8%
and 0.7% lower than our ﬁnal method. The difference between the ﬁnal method and this
second baseline is due to the score recalibration performed by the detection potentials. The
two baselines differ in performance since the second baseline uses a semantic segmentation
(mean IoU of 75.3%) which is better than the other (73.4%) due to its use of detection
potentials. The large gap in their overall instance segmentation performance emphasises
the importance of a good initial semantic segmentation for our method, and underlines the
beneﬁts of a bottom-up approach to instance segmentation.
From the difference in performance between the baseline that does not fully make use of
detection potentials (Row 1 of Table 1), and our ﬁnal method (Row 3 of Table 1), we can
conclude that our higher order detection potentials are an integral part of our system.
4.3 Comparison to other current methods


conclude that our higher order detection potentials are an integral part of our system.
4.3 Comparison to other current methods
In Table 2, we compare our methods to other approaches that have been evaluated on the
same dataset and reported APrresults at multiple IoU thresholds. The SDS method of [13]
was evaluated on the VOC Validation set by [4]. Our approach signiﬁcantly outperforms
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 9
Table 2: Comparison of instance segmentation performance to other methods on the VOC
2012 Validation Set. We report the APrmeasure at ﬁve different IoU thresholds.
Method APrat 0.5 APrat 0.6 APrat 0.7 APrat 0.8 APrat 0.9 APr
vol
SDS [13] 43.8 34.5 21.3 8.7 0.9 Chen et al. [4] 46.3 38.2 27.0 13.5 2.6 PFN [24] 58.7 51.3 42.5 31.2 15.7 52.3
Ours 58.3 52.4 45.4 34.9 20.1 53.1
both [13] and [4] which initially perform object detection, and reﬁne these detections into an


both [13] and [4] which initially perform object detection, and reﬁne these detections into an
instance segmentation. In fact, even our baselines in Table 1 outperform them. The model
of [24] differs from the other two works in that the ﬁrst stage of their pipeline is to also
perform a category-level segmentation of the image. However, their method does not make
use of any information from object detectors, and our approach performs better, particularly
at high IoU thresholds where the difference in APrat an IoU threshold of 0.9 is 4.4%. This
indicates that our instance segmentations are more precise. The fact that our APr
volis only
0.8% greater, despite our APrbeing signiﬁcantly larger at high thresholds suggests that the
method of [24] has a higher APrat lower thresholds (below 0.5, which were not reported by
[24]). This indicates that their method identiﬁes more instances than ours, but the fact that
their APrat higher thresholds is lower than ours also suggests that their segmentations are
not as accurate.
Some success and failure cases of our method are shown in Figure 3. Our method has
trouble differentiating instances which are occluded and also very visually similar to each


Some success and failure cases of our method are shown in Figure 3. Our method has
trouble differentiating instances which are occluded and also very visually similar to each
other (Fig. 3, last two rows). The supplementary material includes a more detailed results
table, and more examples of success and failure cases.
5 Conclusion
We have presented a simple method that effectively leverages state-of-the-art semantic segmentation and object detection networks to perform the less widely studied task of instance
segmentation. Our approach begins with a bottom-up semantic segmentation of the input
image, but is able to reason about instances since the ﬁnal CRF layer in our semantic segmentation network incorporates information from an object detector in the form of a higherorder, detection potential. Our method produces state-of-the-art instance segmentation results, achieving considerable improvements over existing work at high APrthresholds. Our
ﬁnal Instance CRF – which is dynamic in its number of labels – is fully differentiable. This
means that our neural network can be dynamically instantiated for every input image and
trained end-to-end.
Acknowledgements
We thank Bernardino Romera-Paredes, Stuart Golodetz and Shuai Zheng for insightful discussions and feedback. This work was supported by the EPSRC, Clarendon Fund, ERC


We thank Bernardino Romera-Paredes, Stuart Golodetz and Shuai Zheng for insightful discussions and feedback. This work was supported by the EPSRC, Clarendon Fund, ERC
grant ERC-2012-AdG 321162-HELIOS, EPRSRC grant Seebibyte EP/M013774/1 and EPSRC/MURI grant EP/N019474/1.
10 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Input image Object Segmentation Instance Segmentation Ground truth
Figure 3: Some results of our system. Detector outputs are overlayed on the input image. Top
row: An example where our method performs well, handling occlusions and false-positive
detections. Second row: Our method performs well on this easy example which has no
occluding objects of the same class. The naïve method described in §3.3 would have produced the same result. Third row: A failure case where the system is not able to differentiate
visually similar instances. Bottom row: Heavily occluded objects are difﬁcult to identify.
References
[1] Pablo Arbelaez, Jordi Pont-Tuset, Jon Barron, Ferran Marques, and Jitendra Malik.