gradients propagated back through a network vanish, the
credit assignment role of backpropagation is lost, as information about small changes in states in the far past has no
inﬂuence on future states. If gradients explode, gradientbased optimization algorithms struggle to traverse down a
cost surface, because gradient-based optimization assumes
small changes in parameters yield small changes in the objective function. As the number of time steps considered
in the sequence of states grows, the shrinking or expanding
effects associated with the state-to-state transformation at
individual time steps can grow exponentially, yielding respectively vanishing or exploding gradients. See Pascanu
et al. (2010) for a review.
Although the long-term dependencies problem appears
intractable in the absolute (Bengio et al., 1994) for
parametrized dynamical systems, several heuristics have
recently been found to help reduce its effect, such as the
use of self-loops and gating units in the LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014) recurrent architectures. Recent work also supports the idea
of using orthogonal weight matrices to assist optimization
(Saxe et al., 2014; Le et al., 2015).
In this paper, we explore the use of orthogonal and unitary


(Saxe et al., 2014; Le et al., 2015).
In this paper, we explore the use of orthogonal and unitary
matrices in recurrent neural networks. We start in Section 2
by showing a novel bound on the propagated gradients in
recurrent nets when the recurrent matrix is orthogonal. Section 3 discusses the difﬁculties of parameterizing real valued orthogonal matrices and how they can be alleviated by
moving to the complex domain.
We discuss a novel approach to constructing expressive
unitary matrices as the composition of simple unitary matrices which require at most O(nlogn)computation and
O(n)memory, when the state vector has dimension n.
These are unlike general matrices, which require O(n2)
computation and memory. Complex valued representationsarXiv:1511.06464v4  [cs.LG]  25 May 2016
Unitary Evolution Recurrent Neural Networks
have been considered for neural networks in the past, but
with limited success and adoption (Hirose, 2003; Zimmermann et al., 2011). We hope our ﬁndings will change this.
Whilst our model uses complex valued matrices and parameters, all implementation and optimization is possible
with real numbers and has been done in Theano (Bergstra

Summary:
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.


Whilst our model uses complex valued matrices and parameters, all implementation and optimization is possible
with real numbers and has been done in Theano (Bergstra
et al., 2010). This along with other implementation details
are discussed in Section 4, and the code used for the experiments is available online. The potential of the developed
model for learning long term dependencies with relatively
few parameters is explored in Section 5. We ﬁnd that the
proposed architecture generally outperforms LSTMs and
previous approaches based on orthogonal initialization.
2. Orthogonal Weights and Bounding the
Long-Term Gradient
A matrix, W, is orthogonal if W>W=WW>=I.
Orthogonal matrices have the property that they preserve
norm (i.e.kWhk2=khk2) and hence repeated iterative
multiplication of a vector by an orthogonal matrix leaves
the norm of the vector unchanged.
LethTandhtbe the hidden unit vectors for hidden layers
Tandtof a neural network with Thidden layers and T
t. IfCis the objective we are trying to minimize, then
thevanishing andexploding gradient problems refer to the
decay or growth of@C
@htas the number of layers, T, grows.
Letbe a pointwise nonlinearity function, and


decay or growth of@C
@htas the number of layers, T, grows.
Letbe a pointwise nonlinearity function, and
zt+1=Wtht+Vtxt+1
ht+1=(zt+1) (1)
then by the chain rule
@C
@ht=@C
@hT@hT
@ht
=@C
@hTT1Y
k=t@hk+1
@hk=@C
@hTT1Y
k=tDk+1WT
k (2)
where Dk+1=diag(0(zk+1))is the Jacobian matrix of
the pointwise nonlinearity.
In the following we deﬁne the norm of a matrix to refer to
the spectral radius norm (or operator 2-norm) and the norm
of a vector to mean L2-norm. By deﬁnition of the operator norms, for any matrices A;Band vectorvwe have
kAvkk AkkvkandkABkk AkkBk. If the weight
matrices Wkare norm preserving (i.e. orthogonal), thenweprove

@C


matrices Wkare norm preserving (i.e. orthogonal), thenweprove

@C
@ht

=

@C
@hTT1Y
k=tDk+1WT
k

@C
@hT

T1Y
k=t

Dk+1WT
k

=

@C
@hT

T1Y
k=tkDk+1k: (3)
Since Dkis diagonal,kDkk= max j=1;:::;nj0(z(j)
k)j,
withz(j)
kthej-th pre-activation of the k-th hidden layer.
If the absolute value of the derivative 0can take some
value > 1, then this bound is useless, since k@C
@htk

@C
@hT

Ttwhich grows exponentially in T. We therefore cannot effectively bound@C
@htfor deep networks, resulting potentially in exploding gradients.
In the casej0j<  < 1, equation 3 proves that that@C


@htfor deep networks, resulting potentially in exploding gradients.
In the casej0j<  < 1, equation 3 proves that that@C
@httends to 0 exponentially fast as Tgrows, resulting in guaranteed vanishing gradients. This argument makes the rectiﬁed linear unit (ReLU) nonlinearity an attractive choice
(Glorot et al., 2011; Nair & Hinton, 2010). Unless all the
activations are killed at one layer, the maximum entry of
Dkis 1, resulting inkDkk= 1 for all layers k. With
ReLU nonlinearities, we thus have

@C
@ht

@C
@hT

T1Y
k=tkDk+1k=

@C
@hT

: (4)
Most notably, this result holds for a network of arbitrary
depth and renders engineering tricks like gradient clipping
unnecessary (Pascanu et al., 2010).
To the best of our knowledge, this analysis is a novel contribution and the ﬁrst time a neural network architecture has
been mathematically proven to avoid exploding gradients.
3. Unitary Evolution RNNs
Unitary matrices generalize orthogonal matrices to the


been mathematically proven to avoid exploding gradients.
3. Unitary Evolution RNNs
Unitary matrices generalize orthogonal matrices to the
complex domain. A complex valued, norm preserving matrix,U, is called a unitary matrix and is such that UU=
UU=I, where Uis the conjugate transpose of U. Directly parametrizing the set of unitary matrices in such a
way that gradient-based optimization can be applied is not
straightforward because a gradient step will typically yield
a matrix that is not unitary, and projecting on the set of unitary matrices (e.g., by performing an eigendecomposition)
generally costsO(n3)computation when Uisnn.
The most important feature of unitary and orthogonal matrices for our purpose is that they have eigenvalues jwith
Unitary Evolution Recurrent Neural Networks
absolute value 1. The following lemma, proved in (Hoffman & Kunze, 1971), may shed light on a method which
can be used to efﬁciently span a large set of unitary matrices.
Lemma 1. A complex square matrix Wis unitary if and
only if it has an eigendecomposition of the form W=


Lemma 1. A complex square matrix Wis unitary if and
only if it has an eigendecomposition of the form W=
VDV, wheredenotes the conjugate transpose. Here,
V;D2Cnnare complex matrices, where Vis unitary,
andDis a diagonal such that jDj;jj= 1. Furthermore, W
is a real orthogonal matrix if and only if for every eigenvalue Dj;j=jwith eigenvector vj, there is also a complex conjugate eigenvalue k=jwith corresponding
eigenvectorvk=vj.
Writingj=eiwjwithwj2R, a naive method to learn a
unitary matrix would be to ﬁx a basis of eigenvectors V2
Cnnand set
W=VDV; (5)
where Dis a diagonal such that Dj;j=j.
Lemma 1 informs us how to construct a real orthogonal
matrix, W. We must (i) ensure the columns of Vcome
in complex conjugate pairs, vk=vj, and (ii) tie weights


matrix, W. We must (i) ensure the columns of Vcome
in complex conjugate pairs, vk=vj, and (ii) tie weights
wk=wjin order to achieve eiwj=eiwk. Most neural network objective functions are differentiable with respect to the weight matrices, and consequently wjmay be
learned by gradient descent.
Unfortunately the above approach has undesirable properties. Fixing Vand learning wrequiresO
n2
memory,
which is unacceptable given that the number of learned parameters isO(n). Further note that calculating Vufor an
arbitrary vector urequiresO(n2)computation. Setting V
to the identity would satisfy the conditions of the lemma,
whilst reducing memory and computation requirements to
O(n), however, Wwould remain diagonal, and have poor
representation capacity.
We propose an alternative strategy to parameterize unitary
matrices. Since the product of unitary matrices is itself a
unitary matrix, we compose several simple, parameteric,
unitary matrices to construct a single, expressive unitary
matrix. The four unitary building blocks considered are
D, a diagonal matrix with Dj;j=eiwj, with parameterswj2R,
R=I2vv


D, a diagonal matrix with Dj;j=eiwj, with parameterswj2R,
R=I2vv
kvk2, a reﬂection matrix in the complex
vectorv2Cn,
, a ﬁxed random index permutation matrix, and
 F andF1, the Fourier and inverse Fourier transforms.Appealingly, D,Randall permitO(n)storage and
O(n)computation for matrix vector products. FandF1
require no storage and O(nlogn)matrix vector multiplication using the Fast Fourier Transform algorithm. A major
advantage of composing unitary matrices of the form listed
above, is that the number of parameters, memory and computational cost increase almost linearly in the size of the
hidden layer. With such a weight matrix, immensely large
hidden layers are feasible to train, whilst being impossible
in traditional neural networks.
With this in mind, in this work we choose to consider recurrent neural networks with unitary hidden to hidden weight
matrices. Our claim is that the ability to have large hidden
layers where hidden states norms are preserved provides
a powerful tool for modeling long term dependencies in
sequence data. (Bengio et al., 1994) suggest that having
a large memory may be crucial for solving difﬁcult tasks


sequence data. (Bengio et al., 1994) suggest that having
a large memory may be crucial for solving difﬁcult tasks
with long ranging dependencies: the smaller the state dimension, the more information necessarily has to be eliminated when mapping a long sequence to a ﬁxed-dimension
state.
We call any RNN architecture which uses a unitary hidden
to hidden matrix a unitary evolution RNN (uRNN). After
experimenting with several structures, we settled on the following composition
W=D3R2F1D2R 1FD1: (6)
Whilst each but the permutation matrix is complex, we
parameterize and represent them with real numbers for implementation purposes. When the ﬁnal cost is real and differentiable, we may perform gradient descent optimization
to learn the parameters. (Yang et al., 2015) construct a real
valued, non-orthogonal matrix using a similar parameterization with the motivation of parameter reduction by an
order of magnitude on an industrial sized network. This
combined with earlier work (Le et al., 2010) suggests that it
is possible to create highly expressive matrices by composing simple matrices with few parameters. In the following
section, we explain details on how to implement our model
and illustrate how we bypass the potential difﬁculties of
working in the complex domain.


section, we explain details on how to implement our model
and illustrate how we bypass the potential difﬁculties of
working in the complex domain.
4. Architecture details
In this section, we describe the nonlinearity we used, how
we incorporate real valued inputs with complex valued hidden units and map from complex hidden states to real outputs.
4.1. Complex hidden units
Our implementation represents all complex numbers using real values in terms of their real and imaginary parts.
Unitary Evolution Recurrent Neural Networks
Under this framework, we sidestep the lack of support
for complex numbers by most deep learning frameworks.
Consider multiplying the complex weight matrix W=
A+iBby the complex hidden vector h=x+iy,
where A;B;x;y are real. It is trivially true that Wh=
(AxBy) +i(Ay+Bx). When we represent v2Cnas
Re(v)>;Im(v)>>2R2n, we compute complex matrix
vector products with real numbers as follows
Re(Wh)
Im(Wh)
=AB
B ARe(h)
Im(h)
: (7)
More generally, let f:Cn!Cnbe any complex function andz=x+iyany complex vector. We may write


Im(h)
: (7)
More generally, let f:Cn!Cnbe any complex function andz=x+iyany complex vector. We may write
f(z) =(x;y) +i(x;y)where; :Rn!Rn. This
allows us to implement everything using real valued operations, compatible with any any deep learning framework
with automatic differentiation such as Theano.
4.2. Input to Hidden, Nonlinearity, Hidden to Output
As is the case with most recurrent networks, our uRNN follows the same hidden to hidden mapping as equation 1 with
Vt=VandWt=W. Denote the size of the complex
valued hidden states as nh. The input to hidden matrix is
complex valued, V2Cnhnin. We learn the initial hidden
stateh02Cnhas a parameter of the model.
Choosing an appropriate nonlinearity is not trivial in the
complex domain. As discussed in the introduction, using a
ReLU is a natural choice in combination with a norm preserving weight matrix. We ﬁrst experimented with placing
separate ReLU activations on the real and imaginary parts


ReLU is a natural choice in combination with a norm preserving weight matrix. We ﬁrst experimented with placing
separate ReLU activations on the real and imaginary parts
of the hidden states. However, we found that such a nonlinearity usually performed poorly. Our intuition is that applying separate ReLU nonlinearities to the real and imaginary parts brutally impacts the phase of a complex number,
making it difﬁcult to learn structure.
We speculate that maintaining the phase of hidden states
may be important for storing information across a large
number of time steps, and our experiments supported this
claim. A variation of the ReLU that we name modReLU ,
is what we ﬁnally chose. It is a pointwise nonlinearity,
modReLU (z) :C!C, which affects only the absolute
value of a complex number, deﬁned as
modReLU (z) =(jzj+b)z
jzjifjzj+b0
0 ifjzj+b<0(8)
whereb2Ris a bias parameter of the nonlinearity. For a
nhdimensional hidden space we learn nhnonlinearity bias
parameters, one per dimension. Note that the modReLU
is similar to the ReLU in spirit, in fact more concretely


parameters, one per dimension. Note that the modReLU
is similar to the ReLU in spirit, in fact more concretely
modReLU (z) =ReLU (jzj+b)z
jzj.To map hidden states to output, we deﬁne a matrix U2
Rno2nh, wherenois the output dimension. We calculate
a linear output as
ot=URe(ht)
Im(ht)
+bo; (9)
wherebo2Rnois the output bias. The linear output is real
valued (ot2Rno) and can be used for prediction and loss
function calculation akin to typical neural networks (e.g. it
may be passed through a softmax which is used for cross
entropy calculation for classiﬁcation tasks).
4.3. Initialization
Due to the stability of the norm preserving operations of
our network, we found that performance was not very sensitive to initialization of parameters. For full disclosure
and reproducibility, we explain our initialization strategy
for each parameter below.
We initialize VandU(the input and output matrices) as in (Glorot & Bengio, 2010),
with weights sampled independently from uniforms,
Uh
p
6pnin+nout;p
6pnin+nouti
.


with weights sampled independently from uniforms,
Uh
p
6pnin+nout;p
6pnin+nouti
.
The biases,bandboare initialized to 0. This implies
that at initialization, the network is linear with unitary
weights, which seems to help early optimization (Saxe
et al., 2014).
The reﬂection vectors for R1andR2are initialized
coordinate-wise from a uniform U[1;1]. Note that
the reﬂection matrices are invariant to scalar multiplication of the parameter vector, hence the width of the
uniform initialization is unimportant.
The diagonal weights for D1;D2andD3are sampled from a uniform, U[;]. This ensures that the
diagonal entries Dj;jare sampled uniformly over the
complex unit circle.
We initialize h0with a uniform,Uh
q
3
2nh;q
3
2nhi
,
which results in E
kh0k2
= 1. Since the norm of
the hidden units are roughly preserved through unitary
evolution and inputs are typically whitened to have
norm 1, we have hidden states, inputs and linear outputs of the same order of magnitude, which seems to
help optimization.
5. Experiments


norm 1, we have hidden states, inputs and linear outputs of the same order of magnitude, which seems to
help optimization.
5. Experiments
In this section we explore the performance of our uRNN
in relation to (a) RNN with tanh activations, (b) IRNN (Le
et al., 2015), that is an RNN with ReLU activations and
with the recurrent weight matrix initialized to the identity,
Unitary Evolution Recurrent Neural Networks
Figure 1. Results of the copying memory problem for time lags of 100;200;300;500. The LSTM is able to beat the baseline only for
100times steps. Conversely the uRNN is able to completely solve each time length in very few training iterations, without getting stuck
at the baseline.
and (c) LSTM (Hochreiter & Schmidhuber, 1997) models. We show that the uRNN shines quantitatively when
it comes to modeling long term dependencies and exhibits
qualitatively different learning properties to the other models.
We chose a handful of tasks to evaluate the performance
of the various models. The tasks were especially created
to be be pathologically hard, and have been used as benchmarks for testing the ability of a model to capture long-term
memory (Hochreiter & Schmidhuber, 1997; Le et al., 2015;


memory (Hochreiter & Schmidhuber, 1997; Le et al., 2015;
Graves et al., 2014; Martens & Sutskever, 2011)
Of the handful of optimization algorithms we tried on the
various models, RMSProp (Tieleman & Hinton, 2012) lead
to fastest convergence and is what we stuck to for all experiments here on in. However, we found the IRNN to be
particularly unstable; it only ran without blowing up with
incredibly low learning rates and gradient clipping. Since
the performance was so poor relative to other models we
compare against, we do not show IRNN curves in the ﬁgures. In each experiment we use a learning rate of 103
and a decay rate of 0:9. For the LSTM and RNN models,
we had to clip gradients at 1 to avoid exploding gradients.Gradient clipping was unnecessary for the uRNN.
5.1. Copying memory problem
Recurrent networks have been known to have trouble remembering information about inputs seen many time steps
previously (Bengio et al., 1994; Pascanu et al., 2010). We
therefore want to test the uRNN’s ability to recall exactly
data seen a long time ago.
Following a similar setup to (Hochreiter & Schmidhuber,


data seen a long time ago.
Following a similar setup to (Hochreiter & Schmidhuber,
1997), we outline the copy memory task. Consider 10 categories,faig9
i=0. The input takes the form of a T+20 length
vector of categories, where we test over a range of values
ofT. The ﬁrst 10entries are sampled uniformly, independently and with replacement from faig7
i=0, and represent
the sequence which will need to be remembered. The next
T1entries are set to a8, which can be thought of as the
’blank’ category. The next single entry is a9, which represents a delimiter, which should indicate to the algorithm
that it is now required to reproduce the initial 10categories
in the output. The remaining 10entries are set to a8. The
required output sequence consists of T+ 10 repeated entries ofa8, followed by the ﬁrst 10categories of the input
Unitary Evolution Recurrent Neural Networks
Figure 2. Results of the adding problem for T= 100 ;200;400;750. The RNN with tanh is not able to beat the baseline for any time
length. The LSTM and the uRNN show similar performance across time lengths, consistently beating the baseline.


length. The LSTM and the uRNN show similar performance across time lengths, consistently beating the baseline.
sequence in exactly the same order. The goal is to minimize the average cross entropy of category predictions at
each time step of the sequence. The task amounts to having to remember a categorical sequence of length 10, for T
time steps.
A simple baseline can be established by considering an
optimal strategy when no memory is available, which we
deem the memoryless strategy. The memoryless strategy
would be to predict a8forT+ 10 entries and then predict
each of the ﬁnal 10categories from the set faig7
i=0independently and uniformly at random. The categorical cross
entropy of this strategy is10 log(8)
T+20.
We ran experiments where the RNN with tanh activations,
IRNN, LSTM and uRNN had hidden layers of size 80, 80,
40 and 128 respectively. This equates to roughly 6500 parameters per model. In Figure 1, we see that aside from the
simplest case, both the RNN with tanh and more surprisingly the LSTMs get almost exactly the same cost as the
memoryless strategy. This behaviour is consistent with the
results of (Graves et al., 2014), in which poor performance


memoryless strategy. This behaviour is consistent with the
results of (Graves et al., 2014), in which poor performance
is reported for the LSTM for a very similar long term memory problem.
The uRNN consistently achieves perfect performance inrelatively few iterations, even when having to recall sequences after 500 time steps. What is remarkable is that
the uRNN does not get stuck at the baseline at all, whilst
the LSTM and RNN do. This behaviour suggests that the
representations learned by the uRNN have qualitatively different properties from both the LSTM and classical RNNs.
5.2. Adding Problem
We closely follow the adding problem deﬁned in (Hochreiter & Schmidhuber, 1997) to explain the task at hand. Each
input consists of two sequences of length T. The ﬁrst sequence, which we denote x, consists of numbers sampled
uniformly at random U[0;1]. The second sequence is an indicator sequence consisting of exactly two entries of 1 and
remaining entries 0. The ﬁrst 1 entry is located uniformly
at random in the ﬁrst half of the sequence, whilst the second 1 entry is located uniformly at random in the second
half. The output is the sum of the two entries of the ﬁrst sequence, corresponding to where the 1 entries are located in


half. The output is the sum of the two entries of the ﬁrst sequence, corresponding to where the 1 entries are located in
the second sequence. A naive strategy of predicting 1 as the
output regardless of the input sequence gives an expected
mean squared error of 0:167, the variance of the sum of two
independent uniform distributions. This is our baseline to
beat.
Unitary Evolution Recurrent Neural Networks
Figure 3. Results on pixel by pixel MNIST classiﬁcation tasks. The uRNN is able to converge in a fraction of the iterations that the
LSTM requires. The LSTM performs better on MNIST classiﬁcation, but the uRNN outperforms on the more complicated task of
permuted pixels.
We chose to use 128 hidden units for the RNN with tanh,
IRNN and LSTM and 512 for the uRNN. This equates to
roughly 16K parameters for the RNN with tanh and IRNN,
60K for the LSTM and almost 9K for the uRNN. All models were trained using batch sizes of 20 and 50 with the best
results being reported. Our results are shown in Figure 2.
The LSTM and uRNN models are able to convincingly beat
the baseline up to T= 400 time steps. Both models do well


The LSTM and uRNN models are able to convincingly beat
the baseline up to T= 400 time steps. Both models do well
whenT= 750 , but the mean squared error does not reach
close to 0. The uRNN achieves lower test error, but it’s
curve is more noisy. Despite having vastly more parameters, we monitored the LSTM performance to ensure no
overﬁtting.
The RNN with tanh and IRNN were not able to beat the
baseline for any number of time steps. (Le et al., 2015) report that their RNN solve the problem for T= 150 and the
IRNN forT= 300 , but they require over a million iterations before they start learning. Neither of the two models came close to either the uRNN or the LSTM in performance. The stark difference in our ﬁndings are best explained by our use of RMSprop with signiﬁcantly higher
learning rates ( 103as opposed to 108) than (Le et al.,
2015) use for SGD with momentum.
5.3. Pixel-by-pixel MNIST
In this task, suggested by (Le et al., 2015), algorithms are
fed pixels of MNIST (LeCun et al., 1998) sequentially and


In this task, suggested by (Le et al., 2015), algorithms are
fed pixels of MNIST (LeCun et al., 1998) sequentially and
required to output a class label at the end. We consider two
tasks: one where pixels are read in order (from left to right,
bottom to top) and one where the pixels are all randomly
permuted using the same randomly generated permutation
matrix. The same model architectures as for the addingproblem were used for this task, except we now use a softmax for category classiﬁcation. We ran the optimization
algorithms until convergence of the mean categorical cross
entropy on test data, and plot test accuracy in Figure 3.
Both the uRNN and LSTM perform applaudably well here.
On the correct unpermuted MNIST pixels, the LSTM performs better, achieving 98.2 % test accurracy versus 95.1%
for the uRNN. However, when we permute the ordering of
the pixels, the uRNN dominates with 91.4% of accuracy in
contrast to the 88% of the LSTM, despite having less than
a quarter of the parameters. This result is state of the art on
this task, beating the IRNN (Le et al., 2015), which reaches
close to 82% after 1 million training iterations. Notice that


this task, beating the IRNN (Le et al., 2015), which reaches
close to 82% after 1 million training iterations. Notice that
uRNN reaches convergence in less than 20 thousand iterations, while it takes the LSTM from 5 to 10 times as many
to ﬁnish learning.
Permuting the pixels of MNIST images creates many
longer term dependencies across pixels than in the original pixel ordering, where a lot of structure is local. This
makes it necessary for a network to learn and remember
more complicated dependencies across varying time scales.
The results suggest that the uRNN is better able to deal
with such structure over the data, where the LSTM is better
suited to more local sequence structure tasks.
5.4. Exploratory experiments
Norms of hidden state gradients. As discussed in Section 2, key to being able to learn long term dependencies
is in controlling@C
@ht. With this in mind, we explored how
each model propagated gradients, by examining

@C
@ht

as
a function of t. Gradient norms were computed at the beUnitary Evolution Recurrent Neural Networks
Figure 4. From left to right. Norms of the gradients with respect to hidden states i.e.

@C


Figure 4. From left to right. Norms of the gradients with respect to hidden states i.e.

@C
@ht

at (i) beginning of training, (ii) after 100
iterations. (iii) Norms of the hidden states and (iv) L2distance between hidden states and ﬁnal hidden state. The gradient norms of
uRNNs do not decay as fast as for other models as training progresses. uRNN hidden state norms stay much more consistent over time
than the LSTM. LSTM hidden states stay almost the same after a number of time steps, suggesting that it is not able to use new input
information.
ginning of training and again after 100 iterations of training
on the adding problem. The curves are plotted in Figure 4.
It is clear that at ﬁrst, the uRNN propagates gradients perfectly, while each other model has exponentially vanishing
gradients. After 100 iterations of training, each model experiences vanishing gradients, but the uRNN is best able to
propagate information, having much less decay.
Hidden state saturation. We claim that typical recurrent
architectures saturate, in the sense that after they acquire
some information, it becomes much more difﬁcult to acquire further information pertaining to longer dependencies. We took the uRNN and LSTM models trained on


some information, it becomes much more difﬁcult to acquire further information pertaining to longer dependencies. We took the uRNN and LSTM models trained on
the adding problem with T= 200 , and computed a forward pass with newly generated data for the adding problem withT= 1000 . In order to show saturation effects,
we plot the norms of the hidden states and the L2distance
between each state and the last in Figure 4.
In our experiments, it is clear that the uRNN does not suffer
as much as other models do. Notice that whilst the norms of
hidden states in the uRNN grow very steadily over time, in
the LSTM they grow very fast, and then stay constant after
about 500time steps. This behaviour may suggest that the
LSTM hidden states saturate in their ability to incorporate
new information, which is vital for modeling long complicated sequences. It is interesting to see that the LSTM
hidden state at t= 500 , is close to that of t= 1000 , whilst
this is far from the case in the uRNN. Again, this suggests
that the LSTM’s capacity to use new information to alter
its hidden state severly degrades with sequence length. The
uRNN does not suffer from this difﬁculty nearly as badly.
A clear example of this phenomenon was observed in the


uRNN does not suffer from this difﬁculty nearly as badly.
A clear example of this phenomenon was observed in the
adding problem with T= 750 . We found that the Pearson
correlation between the LSTM output prediction and the
ﬁrst of the two uniform samples (whose sum is the target
output) was= 0:991. This suggests that the LSTM learnt
to simply ﬁnd and store the ﬁrst sample, as it was unable
to incorporate any more information by the time it reachedthe second, due to saturation of the hidden states.
6. Discussion
There are a plethora of further ideas that may be explored
from our ﬁndings, both with regards to learning representation and efﬁcient implementation. For example, one hurdle
of modeling long sequences with recurrent networks is the
requirement of storing all hidden state values for the purpose of gradient backpropagation. This can be prohibitive,
since GPU memory is typically a limiting factor of neural
network optimization. However, since our weight matrix is
unitary, its inverse is its conjugate transpose, which is just
as easy to operate with. If further we were to use an invertible nonlinearity function, we would no longer need to store


as easy to operate with. If further we were to use an invertible nonlinearity function, we would no longer need to store
hidden states, since they can be recomputed in the backward pass. This could have potentially huge implications,
as we would be able to reduce memory usage by an order
ofT, the number of time steps. This would make having
immensely large hidden layers possible, perhaps enabling
vast memory representations.
In this paper we demonstrate state of the art performance
on hard problems requiring long term reasoning and memory. These results are based on a novel parameterization
of unitary matrices which permit efﬁcient matrix computations and parameter optimization. Whilst complex domain modeling has been widely succesful in the signal processing community (e.g. Fourier transforms, wavelets), we
have yet to exploit the power of complex valued representation in the deep learning community. Our hope is that
this work will be a step forward in this direction. We motivate the idea of unitary evolution as a novel way to mitigate
the problems of vanishing and exploding gradients. Empirical evidence suggests that our uRNN is better able to pass
gradient information through long sequences and does not
suffer from saturating hidden states as much as LSTMs,
typical RNNs, or RNNs initialized with the identity weight
matrix (IRNNs).


suffer from saturating hidden states as much as LSTMs,
typical RNNs, or RNNs initialized with the identity weight
matrix (IRNNs).
Unitary Evolution Recurrent Neural Networks
Acknowledgments : We thank the developers of
Theano (Bergstra et al., 2010) for their great work.
We thank NSERC, Compute Canada, Canada Research
Chairs and CIFAR for their support. We would also like
to thank C ¸ aglar Gulc ¸ehre, David Krueger, Soroush Mehri,
Marcin Moczulski, Mohammad Pezeshki and Saizheng
Zhang for helpful discussions, comments and code sharing.
References
Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo.
Learning long-term dependencies with gradient descent
is difﬁcult. IEE Transactions on Neural Networks , 5,
1994.
Bergstra, James, Breuleux, Olivier, Bastien, Fr ´ed´eric,
Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua. Theano: a CPU and GPU math expression


compiler. Proceedings of the Python for Scientiﬁc Computing Conference (SciPy) , 2010.
Cho, Kyunghyun, van Merri ¨enboer, Bart, Bahdanau,
Dzmitry, and Bengio, Yoshua. On the properties of neural machine translation: Encoder–Decoder approaches.
InEighth Workshop on Syntax, Semantics and Structure
in Statistical Translation , October 2014.
Collobert, Ronan, Weston, Jason, Bottou, L ´eon, Karlen,
Michael, Kavukcuoglu, Koray, and Kuksa, Pavel. Natural language processing (almost) from scratch. Journal
of Machine Learning Research , 12:2493–2537, 2011.
Glorot, Xavier and Bengio, Yoshua. Understanding the
difﬁculty of training deep feedforward neural networks.
International Conference on Artiﬁcial Intelligence and
Statistics (AISTATS) , 2010.
Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua.
Deep sparse rectiﬁer neural networks. International
Conference on Artiﬁcial Intelligence and Statistics (AISTATS) , 2011.
Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural


Conference on Artiﬁcial Intelligence and Statistics (AISTATS) , 2011.
Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural
turing machines. arXiv preprint arXiv:1410.5401 , 2014.
Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew,
Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara, and
Kingsbury, Brian. Deep neural networks for acoustic
modeling in speech recognition. Signal Processing Magazine , 2012.
Hirose, Akira. Complex-valued neural networks: theories
and applications , volume 5. World Scientiﬁc Publishing
Company Incorporated, 2003.Hochreiter, S. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, T.U. M ¨unich, 1991.
Hochreiter, Sepp and Schmidhuber, J ¨urgen. Long shortterm memory. Neural Computation , 8(9):1735–1780,
1997.
Hoffman, Kenneth and Kunze, Ray. Linear Algebra . Pearson, second edition, 1971.


1997.
Hoffman, Kenneth and Kunze, Ray. Linear Algebra . Pearson, second edition, 1971.
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.
Imagenet classiﬁcation with deep convolutional neural networks. Neural Information Processing Systems ,
2012.
Le, Quoc, Sarl ´os, Tam ´as, and Smola, Alex. Fastfood - approximating kernel expansions in loglinear time. International Conference on Machine Learning , 2010.
Le, Quoc V ., Navdeep, Jaitly, and Hinton, Geoffrey E. A
simple way to initialize recurrent networks of rectiﬁed
linear units. arXiv preprint arXiv:1504.00941 , 2015.
LeCun, Yann, Bottou, L ´eon, Bengio, Yoshua, and Haffner,
Patrick. Gradient-based learning applied to document
recognition. Proceedings of the IEEE , 1998.
Martens, James and Sutskever, Ilya. Learning recurrent
neural networks with hessian-free optimization. International Conference on Machine Learning , 2011.
Nair, Vinod and Hinton, Geoffrey E. Rectiﬁed linear units
improve restricted boltzmann machines. International
Conference on Machine Learning , 2010.


Nair, Vinod and Hinton, Geoffrey E. Rectiﬁed linear units
improve restricted boltzmann machines. International
Conference on Machine Learning , 2010.
Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua.
On the difﬁculty of training recurrent neural networks.
International Conference on Machine Learning , 2010.
Saxe, Andrew M., McLelland, James L., and Ganguli,
Surya. Exact solutions to the nonlinear dynamics of
learning in deep linear neural networks. International
Conference in Learning Representations , 2014.
Tieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5rmsprop: Divide the gradient by a running average of
its recent magnitude. Coursera: Neural Networks for
Machine Learning , 2012.
Yang, Zichao, Moczulski, Marcin, Denil, Misha, de Freitas, Nando, Smola, Alex, Song, Le, and Wang, Ziyu.
Deep fried convnets. International Conference on Computer Vision (ICCV) , 2015.
Zimmermann, Hans-Georg, Minin, Alexey, and
Kusherbaeva, Victoria. Comparison of the complex valued and real valued neural networks trained
with gradient descent and random search algorithms. In
ESANN , 2011.


Higher Order Conditional Random Fields in Deep
Neural Networks
Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, Philip H.S. Torr
University of Oxford
{firstname.lastname }@eng.ox.ac.uk
Abstract. We address the problem of semantic segmentation using deep learning.
Most segmentation systems include a Conditional Random Field (CRF) to produce
a structured output that is consistent with the image’s visual features. Recent deep
learning approaches have incorporated CRFs into Convolutional Neural Networks
(CNNs), with some even training the CRF end-to-end with the rest of the network.
However, these approaches have not employed higher order potentials, which
have previously been shown to signiﬁcantly improve segmentation performance.
In this paper, we demonstrate that two types of higher order potential, based on
object detections and superpixels, can be included in a CRF embedded within a
deep network. We design these higher order potentials to allow inference with
the differentiable mean ﬁeld algorithm. As a result, all the parameters of our
richer CRF model can be learned end-to-end with our pixelwise CNN classiﬁer.
We achieve state-of-the-art segmentation performance on the PASCAL VOC

Summary:
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.

Anurag Arnab
Sadeep Jayasumana
Shuai Zheng
Philip Torr

ECCV 2016

2. Very Deep Convolutional Networks for Large-Scale Image Recognition

3. Fully Convolutional Networks for Semantic Segmentation


We achieve state-of-the-art segmentation performance on the PASCAL VOC
benchmark with these trainable higher order potentials.
Keywords: Semantic Segmentation, Conditional Random Fields, Deep Learning,
Convolutional Neural Networks
1 Introduction
Semantic segmentation involves assigning a visual object class label to every pixel in an
image, resulting in a segmentation with a semantic meaning for each segment. While a
strong pixel-level classiﬁer is critical for obtaining high accuracy in this task, it is also
important to enforce the consistency of the semantic segmentation output with visual
features of the image. For example, segmentation boundaries should usually coincide
with strong edges in the image, and regions in the image with similar appearance should
have the same label.
Recent advances in deep learning have enabled researchers to create stronger classiﬁers, with automatically learned features, within a Convolutional Neural Network
(CNN) [1 –3]. This has resulted in large improvements in semantic segmentation accuracy on widely used benchmarks such as PASCAL VOC [4]. CNN classiﬁers are now
considered the standard choice for pixel-level classiﬁers used in semantic segmentation.
On the other hand, probabilistic graphical models have long been popular for structured prediction of labels, with constraints enforcing label consistency. Conditional

ECCV 2016


On the other hand, probabilistic graphical models have long been popular for structured prediction of labels, with constraints enforcing label consistency. Conditional
Random Fields (CRFs) have been the most common framework, and various rich and
arXiv:1511.08119v4  [cs.CV]  29 Jul 2016

ECCV 2016


Higher Order Conditional Random Fields in Deep Neural Networks 3
We evaluate our higher order potentials on the PASCAL VOC 2012 semantic segmentation benchmark as well as the PASCAL Context dataset, to show signiﬁcant
improvements over our baseline and achieve state-of-the art results.
2 Related Work
Before deep learning became prominent, semantic segmentation was performed with
dense hand-crafted features which were fed into a per-pixel or region classiﬁer [14]. The
individual predictions made by these classiﬁers were often noisy as they lacked global
context, and were thus post-processed with a CRF, making use of prior knowledge such
as the fact that nearby pixels, as well as pixels of similar appearance, are likely to share
the same class label [14, 15].
The CRF model of [14] initially contained only unary and pairwise terms in an
8-neighbourhood, which [16] showed can result in shrinkage bias. Numerous improvements to this model were subsequently proposed including: densely connected pairwise
potentials facilitating interactions between all pairs of image pixels [17], formulating
higher order potentials deﬁned over cliques larger than two nodes [5, 16] in order to
capture more context, modelling co-occurrence of object classes [18 –20], and utilising

ECCV 2016


capture more context, modelling co-occurrence of object classes [18 –20], and utilising
the results of object detectors [6, 21, 22].
Recent advances in deep learning have allowed us to replace hand-crafted features
with features learned speciﬁcally for semantic segmentation. The strength of these
representations was illustrated by [3] who achieved signiﬁcant improvements over
previous hand-crafted methods without using any CRF post-processing. Chen et al. [12]
showed further improvements by post-processing the results of a CNN with a CRF.
Subsequent works [9 –11, 23] have taken this idea further by incorporating a CRF as
layers within a deep network and then learning parameters of both the CRF and CNN
together via backpropagation.
In terms of enhancements to conventional CRF models, Ladicky et al. [6] proposed using an off-the-shelf object detector to provide additional cues for semantic
segmentation. Unlike other approaches that reﬁne a bounding-box detection to produce
a segmentation [8, 24], this method used detector outputs as a soft constraint and can
thus recover from object detection errors. Their formulation, however, used graph-cut
inference, which was only tractable due to the absence of dense pairwise potentials.

ECCV 2016


thus recover from object detection errors. Their formulation, however, used graph-cut
inference, which was only tractable due to the absence of dense pairwise potentials.
Object detectors have also been used by [21, 25], who also modelled variables that
describe the degree to which an object hypothesis is accepted.
We formulate the detection potential in a different manner to [6, 21, 25] so that it is
amenable to mean ﬁeld inference. Mean ﬁeld permits inference with dense pairwise connections, which results in substantial accuracy improvements [10,12, 17]. Furthermore,
mean ﬁeld updates related to our potentials are differentiable and its parameters can thus
be learned in our end-to-end trainable architecture.
We also note that while the semantic segmentation problem has mostly been formulated in terms of pixels [3,10,14], some have expressed it in terms of superpixels [26 –28].
Superpixels can capture more context than a single pixel and computational costs can
also be reduced if one considers pairwise interactions between superpixels rather than individual pixels [21]. However, such superpixel representations assume that the segments
4 Arnab et al.
share boundaries with objects in an image, which is not always true. As a result, several

ECCV 2016


4 Arnab et al.
share boundaries with objects in an image, which is not always true. As a result, several
authors [5, 7] have employed higher order potentials deﬁned over superpixels that encourage label consistency over regions, but do not strictly enforce it. This approach also
allows multiple, non-hierarchical layers of superpixels to be integrated. Our formulation
uses this kind of higher order potential, but in an end-to-end trainable CNN.
Graphical models have been used with CNNs in other areas besides semantic segmentation, such as in pose-estimation [29] and group activity recognition [30]. Alternatively,
Ionescu et al. [31] incorporated structure into a deep network with structured matrix
layers and matrix backpropagation. However, the nature of models used in these works
is substantially different to ours. Some early works that advocated gradient backpropagation through graphical model inference for parameter optimisation include [32, 33]
and [34].
Our work differentiates from the above works since, to our knowledge, we are the ﬁrst
to propose and conduct a thorough experimental investigation of higher order potentials
that are based on detection outputs and superpixel segmentation in a CRF which is
learned end-to-end in a deep network. Note that although [7] formulated mean ﬁeld

ECCV 2016


learned end-to-end in a deep network. Note that although [7] formulated mean ﬁeld
inference with higher order potentials, they did not consider object detection potentials
at all, nor were the parameters learned.
3 Conditional Random Fields
We now review conditional random ﬁelds used in semantic segmentation and introduce
the notation used in the paper. Take an image IwithNpixels, indexed 1,2,...,N . In
semantic segmentation, we attempt to assign every pixel a label from a predeﬁned set of
labels L={l1,l2,...,l L}. Deﬁne a set of random variables X1,X2,...,X N, one for
each pixel, where each Xi∈ L. LetX= [X1X2... X N]T. Any particular assignment
xtoXis thus a solution to the semantic segmentation problem.
We use notations {V}, andV(i)to represent the set of elements of a vector V, and
theithelement of V, respectively. Given a graph Gwhere the vertices are from {X}
and the edges deﬁne connections among these variables, the pair (I,X)is modelled as a

ECCV 2016


and the edges deﬁne connections among these variables, the pair (I,X)is modelled as a
CRF characterised by Pr(X=x|I) = (1/Z(I)) exp( −E(x|I)), whereE(x|I)is the
energy of the assignment xandZ(I)is the normalisation factor known as the partition
function. We drop the conditioning on Ihereafter to keep the notation uncluttered. The
energyE(x)of an assignment is deﬁned using the set of cliques Cin the graph G. More
speciﬁcally, E(x) =/summationtext
c∈Cψc(xc),where xcis a vector formed by selecting elements
ofxthat correspond to random variables belonging to the clique c, andψc(.)is the
cost function for the clique c. The function, ψc(.), usually uses prior knowledge about a
good segmentation, as well as information from the image, the observation the CRF is
conditioned on.
Minimising the energy yields the maximum a posteriori (MAP) labelling of the
image i.e.the most probable label assignment given the observation (image). When
dense pairwise potentials are used in the CRF to obtain higher accuracy, exact inference

ECCV 2016


image i.e.the most probable label assignment given the observation (image). When
dense pairwise potentials are used in the CRF to obtain higher accuracy, exact inference
is impracticable, and one has to resort to an approximate inference method such as mean
ﬁeld inference [17]. Mean ﬁeld inference is particularly appealing in a deep learning
setting since it is possible to formulate it as a Recurrent Neural Network [10].
Higher Order Conditional Random Fields in Deep Neural Networks 5
4 CRF with Higher Order Potentials
Many CRF models that have been incorporated into deep learning frameworks [10, 12]
have so far used only unary and pairwise potentials. However, potentials deﬁned on
higher order cliques have been shown to be useful in previous works such as [7, 16]. The
key contribution of this paper is to show that a number of explicit higher order potentials
can be added to CRFs to improve image segmentation, while staying compatible with
deep learning. We formulate these higher order potentials in a manner that mean ﬁeld
inference can still be used to solve the CRF. Advantages of mean ﬁeld inference are
twofold: First, it enables efﬁcient inference when using densely-connected pairwise

ECCV 2016


twofold: First, it enables efﬁcient inference when using densely-connected pairwise
potentials. Multiple works, [10,33] have shown that dense pairwise connections result in
substantial accuracy improvements, particularly at image boundaries [12, 17]. Secondly,
we keep all our mean ﬁeld updates differentiable with respect to their inputs as well
as the CRF parameters introduced. This design enables us to use backpropagation to
automatically learn all the parameters in the introduced potentials.
We use two types of higher order potential, one based on object detections and the
other based on superpixels. These are detailed in Sections 4.1 and 4.2 respectively. Our
complete CRF model is represented by
E(x) =/summationdisplay
iψU
i(xi) +/summationdisplay
i < jψP
ij(xi,xj) +/summationdisplay
dψDet
d(xd) +/summationdisplay
sψSP
s(xs),(1)
where the ﬁrst two terms ψU
i(.)andψP
ij(.,.)are the usual unary and densely-connected
pairwise energies [17] and the last two terms are the newly introduced higher order

ECCV 2016


i(.)andψP
ij(.,.)are the usual unary and densely-connected
pairwise energies [17] and the last two terms are the newly introduced higher order
energies. Energies from the object detection take the form ψDet
d(xd), where vector xd
is formed by elements of xthat correspond to the foreground pixels of the dthobject
detection. Superpixel label consistency based energies take the form ψSP
s(xs), where xs
is formed by elements of xthat correspond to the pixels belonging to the sthsuperpixel.
4.1 Object Detection Based Potentials
Semantic segmentation errors can be classiﬁed into two broad categories [35]: recognition
and boundary errors. Boundary errors occur when semantic labels are incorrect at the
edges of objects, and it has been shown that densely connected CRFs with appearanceconsistency terms are effective at combating this problem [17]. On the other hand,
recognition errors occur when object categories are recognised incorrectly or not at
all. A CRF with only unary and pairwise potentials cannot effectively correct these
errors since they are caused by poor unary classiﬁcation. However, we propose that a
state-of-the-art object detector [36,37] capable of recognising and localising objects, can

ECCV 2016


state-of-the-art object detector [36,37] capable of recognising and localising objects, can
provide important information in this situation and help reduce the recognition error, as
shown in Fig. 2.
A key challenge in feeding-in object-detection potentials to semantic segmentation
are false detections. A na ¨ıve approach of adding an object detector’s output to a CRF
formulated to solve the problem of semantic segmentation would confuse the CRF
due to the presence of the false positives in the detector’s output. Therefore, a robust
formulation, which can automatically reject object detection false positives when they

ECCV 2016


Higher Order Conditional Random Fields in Deep Neural Networks 7
positive: some of the detections obtained from the object detector may be identiﬁed to
be false following CRF inference.
AllYdvariables are added to the CRF which previously contained only Xivariables.
Let each (Xd,Yd), where {Xd}={Xi∈ {X}|i∈Fd}, form a clique cdin the
CRF. We deﬁne the detection-based higher order energy associated with a particular
assignment (xd,yd)to the clique (Xd,Yd)as follows:
ψDet
d(Xd=xd,Yd=yd) =

wDetsd
nd/summationtextnd
i=1[x(i)
d=ld]ifyd= 0,
wDetsd
nd/summationtextnd
i=1[x(i)
d/ne}ationslash=ld]ifyd= 1,(2)
wherend=|Fd|is the number of foreground pixels in the dthdetection,x(i)
dis theith
element of the vector xd,wDetis a learnable weight parameter, and [.]is the Iverson

ECCV 2016


dis theith
element of the vector xd,wDetis a learnable weight parameter, and [.]is the Iverson
bracket. Note that this potential encourages X(i)
ds to take the value ldwhenYdis1, and
at the same time encourages Ydto be 0when manyX(i)
ds do not take ld. In other words,
it enforces the consistency among X(i)
ds andYd.
An important property of the above deﬁnition of ψDet
d(.)is that it can be simpliﬁed
as a sum of pairwise potentials between Ydand eachX(i)
dfori= 1,2,...,n d. That is,
ψDet
d(Xd=xd,Yd=yd) =nd/summationdisplay
i=1fd(x(i)
d,yd), where,
fd(x(i)
d,yd) =

wDetsd
nd[x(i)
d=ld] ifyd= 0,
wDetsd
nd[x(i)
d/ne}ationslash=ld] ifyd= 1.(3)

ECCV 2016


d=ld] ifyd= 0,
wDetsd
nd[x(i)
d/ne}ationslash=ld] ifyd= 1.(3)
We make use of this simpliﬁcation in Section 5 when deriving the mean ﬁeld updates
associated with this potential.
For the latent Yvariables, in addition to the joint potentials with Xvariables,
described in Eq. (2)and(3), we also include unary potentials, which are initialised
from the score sdof the object detection. The underlying idea is that if the object
detector detects an object with high conﬁdence, the CRF in turn starts with a high initial
conﬁdence about the validity of that detection. This conﬁdence can, of course, change
during the CRF inference depending on other information ( e.g.segmentation unary
potentials) available to the CRF.
Examples of input images with multiple detections and GrabCut foreground masks
are shown in Figure 3. Note how false detections are ignored and erroneous parts of the
foreground mask are also largely ignored.
4.2 Superpixel Based Potentials
The next type of higher order potential we use is based on the idea that superpixels

ECCV 2016


foreground mask are also largely ignored.
4.2 Superpixel Based Potentials
The next type of higher order potential we use is based on the idea that superpixels
obtained from oversegmentation [39, 40] quite often contain pixels from the same visual
object. It is therefore natural to encourage pixels inside a superpixel to have the same
semantic label. Once again, this should not be a hard constraint in order to keep the
8 Arnab et al.
(a) (b) (c) (d)
Fig. 3: Effects of imperfect foreground segmentation (a,b) Detected objects, as well
as the foreground masks obtained from GrabCut. (c,d) Output using detection potentials.
Incorrect parts of the foreground segmentation of the main aeroplane, and entire TV
detection have been ignored by CRF inference as they did not agree with the other energy
terms. The person is a failure case though as the detection has caused part of the sofa to
be erroneously labelled.
(a) (b) (c)
Fig. 4: Segmentation enhancement from superpixel based potentials (a) The output
of our system without any superpixel potentials. (b) Superpixels obtained from the image
using the method of [39]. Only one “layer” of superpixels is shown. In practice, we

ECCV 2016


using the method of [39]. Only one “layer” of superpixels is shown. In practice, we
used four. (c) The output using superpixel potentials. The result has improved as we
encourage consistency over superpixel regions. This removes some of the spurious noise
that was present previously.
algorithm robust to initial superpixel segmentation errors and to violations of this key
assumption.
We use two types of energies in the CRF to encourage superpixel consistency in
semantic segmentation. Firstly, we use the Pn-Potts model type energy [41], which is
described by,
ψSP
s(Xs=xs) =

wLow(l) if allx(i)
s=l,
wHigh otherwise,(4)
wherewLow(l)<w High for alll, and{Xs} ⊂ {X}is a clique deﬁned by a superpixel.
The primary idea is that assigning different labels to pixels in the same superpixel incurs
a higher cost, whereas one obtains a lower cost if the labelling is consistent throughout
the superpixel. Costs wLow(l)andwHigh are learnable during the end-to-end training of
the network.

ECCV 2016


the superpixel. Costs wLow(l)andwHigh are learnable during the end-to-end training of
the network.
Secondly, to make this potential stronger, we average initial unary potentials from the
classiﬁer (the CNN in our case), across all pixels in the superpixel and use the average
Higher Order Conditional Random Fields in Deep Neural Networks 9
as an additional unary potential for those pixels. During experiments, we observed that
superpixel based higher order energy helps in getting rid of small spurious regions of
wrong labels in the segmentation output, as shown in Fig. 4.
5 Mean Field Updates and Their Differentials
This section discusses the mean ﬁeld updates for the higher order potentials previously
introduced. These update operations are differentiable with respect to the Qi(Xi)distribution inputs at each iteration, as well as the parameters of our higher order potentials.
This allows us to train our CRF end-to-end as another layer of a neural network.
Take a CRF with random variables V1,V2,...,V Nand a set of cliques C, which
includes unary, pairwise and higher order cliques. Mean ﬁeld inference approximates
the joint distribution Pr(V=v)with the product of marginals/producttext
iQ(Vi=vi). We

ECCV 2016


the joint distribution Pr(V=v)with the product of marginals/producttext
iQ(Vi=vi). We
useQ(Vc=vc)to denote the marginal probability mass for a subset {Vc}of these
variables. Where there is no ambiguity, we use the short-hand notation Q(vc)to represent
Q(Vc=vc). General mean ﬁeld updates of such a CRF take the form [13]
Qt+1(Vi=v) =1
Ziexp
−/summationdisplay
c∈C/summationdisplay
{vc|vi=v}Qt(vc−i)ψc(vc)
, (5)
whereQtis the marginal after the tthiteration, vcan assignment to all variables in
cliquec,vc−ian assignment to all variables in cexcept forVi,ψc(vc)is the cost of
assigning vcto the clique c, andZiis the normalisation constant that makes Q(Vi=v)
a probability mass function after the update.
Updates from Detection Based Potentials Following Eq. (3)above, we now use
Eq.(5)to derive the mean ﬁeld updates related to ψDet

ECCV 2016


Updates from Detection Based Potentials Following Eq. (3)above, we now use
Eq.(5)to derive the mean ﬁeld updates related to ψDet
d. The contribution from ψDet
dto
the update of Q(X(i)
d=l)takes the form
/summationdisplay
{(xd,yd)|x(i)
d=l}Q(xd−i,yd)ψDet
d(xd,yd) =

wDetsd
ndQ(Yd= 0) ifl=ld,
wDetsd
ndQ(Yd= 1) otherwise,(6)
where xd−iis an assignment to Xdwith theithelement deleted. Using the same equations, we derive the contribution from the energy ψDet
dto the update of Q(Yd=b)to
take the form
/summationdisplay
{(xd,yd)|yd=b}Q(xd)ψDet
d(xd,yd) =

wDetsd
nd/summationtextnd
i=1Q(X(i)
d=ld) ifb= 0,
wDetsd
nd/summationtextnd

ECCV 2016


nd/summationtextnd
i=1Q(X(i)
d=ld) ifb= 0,
wDetsd
nd/summationtextnd
i=1(1−Q(X(i)
d=ld))otherwise.
(7)
It is possible to increase the number of parameters in ψDet
d(.). Since we use backpropagation to learn these parameters automatically during end-to-end training, it is desirable
to have a high number of parameters to increase the ﬂexibility of the model. Following
10 Arnab et al.
this idea, we made the weight wDetclass speciﬁc, that is, a function wDet(ld)is used
instead ofwDetin Eqs. (2),(6)and (7). The underlying assumption is that detector
outputs can be very helpful for certain classes, while being not so useful for classes that
the detector performs poorly on, or classes for which foreground segmentation is often
inaccurate.
Note that due to the presence of detection potentials in the CRF, error differentials
calculated with respect to the Xvariable unary potentials and pairwise parameters will
no longer be valid in the forms described in [10]. The error differentials with respect to

ECCV 2016


calculated with respect to the Xvariable unary potentials and pairwise parameters will
no longer be valid in the forms described in [10]. The error differentials with respect to
theXandYvariables, as well as class-speciﬁc detection potential weights wDet(l)are
included in the supplementary material.
Updates for Superpixel Based Potentials The contribution from the Pn-Potts type
potential to the mean ﬁeld update of Q(xi=l), where pixel iis in the superpixel clique
s, was derived in [7] as
/summationdisplay
{xs|x(i)
s=l}Q(xs−i)ψSP
s(xs) =wLow(l)/productdisplay
j∈c,j/ne}ationslash=iQ(Xj=l)+wHigh
1−/productdisplay
j∈c−iQ(Xj=l)
.
(8)
This update operation is differentiable with respect to the parameters wLow(l)andwHigh,
allowing us to optimise them via backpropagation, and also with respect to the Q(X)
values enabling us to optimise previous layers in the network.

ECCV 2016


allowing us to optimise them via backpropagation, and also with respect to the Q(X)
values enabling us to optimise previous layers in the network.
Convergence of parallel mean ﬁeld updates Mean ﬁeld with parallel updates, as
proposed in [17] for speed, does not have any convergence guarantees in the general case.
However, we usually empirically observed convergence with higher order potentials,
without damping the mean ﬁeld update as described in [7, 42]. This may be explained by
the fact that the unaries from the initial pixelwise-prediction part of our network provide
a good initialisation. In cases where the mean ﬁeld energy did not converge, we still
empirically observed good ﬁnal segmentations.
6 Experiments
We evaluate our new CRF formulation on two different datasets using the CRF-RNN
network [10] as the main baseline, since we are essentially enriching the CRF model
of [10]. We then present ablation studies on our models.
6.1 Experimental set-up and results
Our deep network consists of two conceptually different, but jointly trained stages. The
ﬁrst, “unary” part of our network is formed by the FCN-8s architecture [3]. It is initialised

ECCV 2016


ﬁrst, “unary” part of our network is formed by the FCN-8s architecture [3]. It is initialised
from the Imagenet-trained VGG-16 network [2], and then ﬁne-tuned with data from
the VOC 2012 training set [4], extra VOC annotations of [43] and the MS COCO [44]
dataset.
Higher Order Conditional Random Fields in Deep Neural Networks 11
Table 1: Comparison of each higher order
potential with baseline on VOC 2012 reduced validation set
MethodReduced
val set(%)
Baseline (unary + pairwise) [10] 72.9
Superpixels only 74.0
Detections only 74.9
Detections and Superpixels 75.8Table 2: Mean IoU accuracy on VOC 2012
test set. All methods are trained with MS
COCO [44] data
Method Test set(%)
Ours 77.9
DPN [9] 77.5
Centrale Super Boundaries [45] 75.7
Dilated Convolutions [46] 75.3
BoxSup [35] 75.2
DeepLab Attention [47] 75.1
CRF-RNN (baseline) [10] 74.7
DeepLab WSSL [48] 73.9

ECCV 2016


DeepLab Attention [47] 75.1
CRF-RNN (baseline) [10] 74.7
DeepLab WSSL [48] 73.9
DeepLab [12] 72.7
Table 3: Mean Intersection over Union (IoU) results on PASCAL Context validation set
compared to other current methods.
Method Ours BoxSup [35] ParseNet [49] CRF-RNN [10] FCN-8s [3] CFM [28]
Mean IoU (%) 41.3 40.5 40.4 39.3 37.8 34.4
The output of the ﬁrst stage is fed into our CRF inference network. This is implemented using the mean ﬁeld update operations and their differentials described in
Section 5. Five iterations of mean ﬁeld inference were performed during training. Our
CRF network has two additional inputs in addition to segmentation unaries obtained from
the FCN-8s network: data from the object detector and superpixel oversegmentations of
the image.
We used the publicly available code and model of the Faster R-CNN [37] object detector. The fully automated version of GrabCut [38] was then used to obtain foregrounds
from the detection bounding boxes. These choices were made after conducting preliminary experiments with alternate detection and foreground segmentation algorithms.

ECCV 2016


from the detection bounding boxes. These choices were made after conducting preliminary experiments with alternate detection and foreground segmentation algorithms.
Four levels of superpixel oversegmentations were used, with increasing superpixel
size to deﬁne the cliques used in this potential. Four levels were used since performance
on the VOC validation set stopped increasing after this number. We used the superpixel
method of [39] as it was shown to adhere to object boundaries the best [40], but our
method generalises to any oversegmentation algorithm.
We trained the full network end-to-end, optimising the weights of the CNN classiﬁer
(FCN-8s) and CRF parameters jointly. We initialised our network using the publicly
available weights of [10], and trained with a learning rate of 10−10and momentum of
0.99. The learning rate is low because the loss was not normalised by the number of
pixels in the training image. This is to have a larger loss for images with more pixels.
When training our CRF, we only used VOC 2012 data [4] as it has the most accurate
labelling, particularly around boundaries.
12 Arnab et al.
PASCAL VOC 2012 Dataset The improvement obtained by each higher order potential
was evaluated on the same reduced validation set [3] used by our baseline [10]. As Table

ECCV 2016


PASCAL VOC 2012 Dataset The improvement obtained by each higher order potential
was evaluated on the same reduced validation set [3] used by our baseline [10]. As Table
1 shows, each new higher order potential improves the mean IoU over the baseline. We
only report test set results for our best method since the VOC guidelines discourage
the use of the test set for ablation studies. On the test set (Table 2), we outperform our
baseline by 3.2% which equates to a 12.6% reduction in the error rate. This sets a new
state-of-the-art on the VOC dataset. Qualitative results highlighting success and failure
cases of our algorithm, as well as more detailed results, are shown in our supplementary
material.
PASCAL Context Table 3 shows our state-of-the-art results on the recently released
PASCAL Context dataset [50]. We trained on the provided training set of 4998 images,
and evaluated on the validation set of 5105 images. This dataset augments VOC with
annotations for all objects in the scene. As a result, there are 59 classes as opposed to
the 20 in the VOC dataset. Many of these new labels are “stuff” classes such as “grass”
and “sky”. Our object detectors are therefore only trained for 20 of the 59 labels in this

ECCV 2016


and “sky”. Our object detectors are therefore only trained for 20 of the 59 labels in this
dataset. Nevertheless, we improve by 0.8% over the previous state-of-the-art [35] and
2% over our baseline [10].
6.2 Ablation Studies
We perform additional experiments to determine the errors made by our system, show
the beneﬁts of end-to-end training and compare our detection potentials to a simpler
baseline. Unless otherwise stated, these experiments are performed on the VOC 2012
reduced validation set.
Error Analysis To analyse the improvements made by our higher order potentials, we
separately evaluate the performance on the “boundary” and “interior” regions in a similar
manner to [35]. As shown in Fig. 5 c) and d), we consider a narrow band (trimap [16])
around the “void” labels annotated in the VOC 2012 reduced validation set. The mean
IoU of pixels lying within this band is termed the “Boundary IoU” whilst the “Interior
IoU” is evaluated outside this region.
Fig. 5 shows our results as the trimap width is varied. Adding the detection potentials

ECCV 2016


IoU” is evaluated outside this region.
Fig. 5 shows our results as the trimap width is varied. Adding the detection potentials
improves the Interior IoU over our baseline (only pairwise potentials [10]) as the object
detector may recognise objects in the image which the pixelwise classiﬁcation stage of
our network may have missed out. However, the detection potentials also improve the
Boundary IoU for all tested trimap widths as well. Improving the recognition of pixels
in the interior of an object also helps with delineating the boundaries since the strength
of the pairwise potentials exerted by the Qdistributions at each of the correctly-detected
pixels increase.
Our superpixel priors also increase the Interior IoU with respect to the baseline.
Encouraging consistency over regions helps to get rid of spurious regions of wrong
labels (as shown in Fig. 4). Fig. 5 suggests that most of this improvement occurs in the
interior of an object. The Boundary IoU is slightly lower than the baseline, and this may
Higher Order Conditional Random Fields in Deep Neural Networks 13
a) Image (b) Ground truth (c) Boundary (d) Interior
0 5 10 15 20 25 30 35 406668707274767880
Trimap WidthInterior IoU

Pairwise only

ECCV 2016


0 5 10 15 20 25 30 35 406668707274767880
Trimap WidthInterior IoU

Pairwise only
Superpixels
Detections
Detections and Superpixels
0 5 10 15 20 25 30 35 40505560657075
Trimap WidthBoundary IoU

Pairwise only
Superpixels
Detections
Detections and Superpixels
(e) Interior IoU (f) Boundary IoU
Fig. 5: Error analysis on VOC 2012 reduced validation set The IoU is computed for
boundary and interior regions for various trimap widths. An example of the Boundary
and Interior regions for a sample image using a width of 9 pixels is shown in white in
the top row. Black regions are ignored in the IoU calculation.
be due to the fact that superpixels do not always align correctly with the edges of an
object (the “boundary recall” of various superpixel methods are evaluated in [40]).
We can see that the combination of detection and superpixel potentials results in a
substantial improvement in our Interior IoU. This is the primary reason our overall IoU
on the VOC benchmark increases with higher order potentials.
Beneﬁts of end-to-end training Table 4 shows how end-to-end training outperforms

ECCV 2016


on the VOC benchmark increases with higher order potentials.
Beneﬁts of end-to-end training Table 4 shows how end-to-end training outperforms
piecewise training. We trained the CRF piecewise by freezing the weights of the unary
part of the network, and only learning the CRF parameters.
Our results in Table 2 used the FCN-8s [3] architecture to generate unaries. To show
that our higher order potentials improve performance regardless of the underlying CNN
used for producing unaries, we also perform an experiment using our reimplementation
Table 4: Comparison of mean IoU (%) obtained on VOC 2012 reduced validation set
from end-to-end and piecewise training
Method FCN-8s DCN
Unary only, ﬁne-tuned on COCO 68.3 68.6
Pairwise CRF trained piecewise 69.5 70.7
Pairwise CRF trained end-to-end 72.9 72.5
Higher Order CRF trained piecewise 73.6 73.5
Higher Order CRF trained end-to-end 75.8 75.0
Test set performance of best model 77.9 76.9
14 Arnab et al.
of the “front-end” module proposed in the Dilated Convolution Network (DCN) of [46]

ECCV 2016


14 Arnab et al.
of the “front-end” module proposed in the Dilated Convolution Network (DCN) of [46]
instead of FCN-8s.
Table 4 shows that end-to-end training of the CRF yields considerable improvements
over piecewise training. This was the case when using either FCN-8s or DCN for
obtaining the initial unaries before performing CRF inference with higher order potentials.
This suggests that our CRF network module can be plugged into different architectures
and achieve performance improvements.
Baseline for detections To evaluate the efﬁcacy of our detection potentials, we formulate a simpler baseline since no other methods use detection information at inference
time (BoxSup [35] derives ground truth for training using ground-truth bounding boxes).
Our baseline is similar to CRF-RNN [10], but prior to CRF inference, we take the
segmentation mask from the object detection and add a unary potential proportional to
the detector’s conﬁdence to the unary potentials for those pixels. We then perform meanﬁeld inference (with only pairwise terms) on these “augmented” unaries. Using this

ECCV 2016


method, the mean IoU increases from 72.9% to 73.6%, which is signiﬁcantly less than
the 74.9% which we obtained using only our detection potentials without superpixels
(Table 1).
Our detection potentials perform better since our latent Ydetection variables model
whether the detection hypothesis is accepted or not. Our CRF inference is able to evaluate
object detection inputs in light of other potentials. Inference increases the relative score
of detections which agree with the segmentation, and decreases the score of detections
which do not agree with other energies in the CRF. Figures 2 b) and d) show examples
of false-positive detections that have been ignored and correct detections that have been
used to reﬁne our segmentation. Our baseline, on the other hand, is far more sensitive to
erroneous detections as it cannot adjust the weight given to them during inference.
7 Conclusion
We presented a CRF model with two different higher order potentials to tackle the
semantic segmentation problem. The ﬁrst potential is based on the intuitive idea that
object detection can provide useful cues for semantic segmentation. Our formulation is
capable of automatically rejecting false object detections that do not agree at all with
the semantic segmentation. Secondly, we used a potential that encourages superpixels to

ECCV 2016


capable of automatically rejecting false object detections that do not agree at all with
the semantic segmentation. Secondly, we used a potential that encourages superpixels to
have consistent labelling. These two new potentials can co-exist with the usual unary
and pairwise potentials in a CRF.
Importantly, we showed that efﬁcient mean ﬁeld inference is still possible in the
presence of the new higher order potentials and derived the explicit forms of the mean
ﬁeld updates and their differentials. This enabled us to implement the new CRF model
as a stack of CNN layers and to train it end-to-end in a uniﬁed deep network with a
pixelwise CNN classiﬁer. We experimentally showed that the addition of higher order
potentials results in a signiﬁcant increase in semantic segmentation accuracy allowing us
to reach state-of-the-art performance.
This work was supported by ERC grant ERC-2012-AdG 321162-HELIOS, EPSRC grant
Seebibyte EP/M013774/1, EPSRC/MURI grant EP/N019474/1 and the Clarendon Fund.
Higher Order Conditional Random Fields in Deep Neural Networks 15
References

ECCV 2016


Higher Order Conditional Random Fields in Deep Neural Networks 15
References
1.Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classiﬁcation with deep convolutional
neural networks. In: NIPS. (2012) 1097–1105
2.Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image
recognition. In: ICLR. (2015)
3.Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation.
In: CVPR. (2015)
4.Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual
object classes (voc) challenge. IJCV (2010)
5.Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Associative hierarchical crfs for object class
image segmentation. In: ICCV . (2009) 739–746
6.Ladick ´y, L., Sturgess, P., Alahari, K., Russell, C., Torr, P.H.S.: What, where and how many?

ECCV 2016


combining object detectors and crfs. In: ECCV . (2010) 424–437
7.Vineet, V., Warrell, J., Torr, P.H.: Filter-based mean-ﬁeld inference for random ﬁelds with
higher-order terms and product label-spaces. IJCV (2014)
8.Hariharan, B., Arbel ´aez, P., Girshick, R., Malik, J.: Simultaneous detection and segmentation.
In: ECCV . Springer (2014) 297–312
9.Liu, Z., Li, X., Luo, P., Loy, C.C., Tang, X.: Semantic image segmentation via deep parsing
network. In: ICCV . (2015)
10.Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr,
P.: Conditional random ﬁelds as recurrent neural networks. In: ICCV . (2015)
11.Lin, G., Shen, C., Reid, I.: Efﬁcient piecewise training of deep structured models for semantic
segmentation. In: CVPR. (2016)

ECCV 2016


segmentation. In: CVPR. (2016)
12.Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR (2015)
13.Koller, D., Friedman, N.: Probabilistic graphical models: principles and techniques. MIT
press (2009)
14.Shotton, J., Winn, J., Rother, C., Criminisi, A.: Textonboost for image understanding: Multiclass object recognition and segmentation by jointly modeling texture, layout, and context.
IJCV (2009)
15.He, X., Zemel, R.S., Carreira-Perpi ˜n´an, M. ´A.: Multiscale conditional random ﬁelds for image
labeling. In: CVPR, IEEE (2004)
16.Kohli, P., Ladicky, L., Torr, P.: Robust higher order potentials for enforcing label consistency.
IJCV 82(3) (2009) 302–324
17.Kr¨ahenb ¨uhl, P., Koltun, V.: Efﬁcient inference in fully connected CRFs with Gaussian edge

ECCV 2016


17.Kr¨ahenb ¨uhl, P., Koltun, V.: Efﬁcient inference in fully connected CRFs with Gaussian edge
potentials. In: NIPS. (2011)
18.Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Graph cut based inference with co-occurrence
statistics. In: ECCV . (2010) 239–253
19.Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., Belongie, S.: Objects in context.
In: ICCV . (2007) 1–8
20.Gonfaus, J.M., Boix, X., Van de Weijer, J., Bagdanov, A.D., Serrat, J., Gonzalez, J.: Harmony
potentials for joint classiﬁcation and segmentation. In: CVPR, IEEE (2010) 3280–3287
21.Yao, J., Fidler, S., Urtasun, R.: Describing the Scene as a Whole: Joint Object Detection,
Scene Classiﬁcation and Semantic Segmentation. In: CVPR. (2012) 702–709

ECCV 2016


Scene Classiﬁcation and Semantic Segmentation. In: CVPR. (2012) 702–709
22.Wojek, C., Schiele, B.: A dynamic conditional random ﬁeld model for joint labeling of object
and scene classes. In: ECCV , Springer (2008) 733–747
23.Lin, G., Shen, C., Reid, I., van den Hengel, A.: Deeply learning the messages in message
passing inference. In: NIPS. (2015) 361–369
16 Arnab et al.
24.Yang, Y., Hallman, S., Ramanan, D., Fowlkes, C.C.: Layered object models for image
segmentation. PAMI (2012)
25.Sun, M., Kim, B.s., Kohli, P., Savarese, S.: Relating things and stuff via objectproperty
interactions. PAMI 36(7) (2014) 1370–1383
26.Carreira, J., Caseiro, R., Batista, J., Sminchisescu, C.: Semantic segmentation with secondorder pooling. In: ECCV . (2012) 430–443
27.Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene

ECCV 2016


27.Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene
labeling. PAMI (2013)
28.Dai, J., He, K., Sun, J.: Convolutional feature masking for joint object and stuff segmentation.
CVPR (2015)
29.Tompson, J.J., Jain, A., LeCun, Y ., Bregler, C.: Joint training of a convolutional network and
a graphical model for human pose estimation. In: NIPS. (2014) 1799–1807
30.Deng, Z., Zhai, M., Chen, L., Liu, Y., Muralidharan, S., Roshtkhari, M.J., Mori, G.: Deep
structured models for group activity recognition. In: BMVC. (2015)
31.Ionescu, C., Vantzos, O., Sminchisescu, C.: Matrix backpropagation for deep networks with
structured layers. In: ICCV . (2015) 2965–2973
32.Domke, J.: Learning graphical model parameters with approximate marginal inference. PAMI
(2013)
33.Kr¨ahenb ¨uhl, P., Koltun, V.: Parameter learning and convergent inference for dense random

ECCV 2016


(2013)
33.Kr¨ahenb ¨uhl, P., Koltun, V.: Parameter learning and convergent inference for dense random
ﬁelds. In: ICML. (2013)
34.Ross, S., Munoz, D., Hebert, M., Bagnell, J.A.: Learning message-passing inference machines
for structured prediction. In: CVPR. (2011)
35.Dai, J., He, K., Sun, J.: Boxsup: Exploiting bounding boxes to supervise convolutional
networks for semantic segmentation. In: ICCV . (2015)
36. Girshick, R.: Fast r-cnn. In: ICCV . (2015)
37.Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with
region proposal networks. In: NIPS. (2015)
38.Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using
iterated graph cuts. ACM TOG (2004)
39.Felzenszwalb, P.F., Huttenlocher, D.P.: Efﬁcient graph-based image segmentation. IJCV
(2004)

ECCV 2016


(2004)
40.Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: Slic superpixels compared
to state-of-the-art superpixel methods. PAMI 34(11) (2012) 2274–2282
41. Kohli, P., Kumar, M.P., Torr, P.H.: P3 & beyond: Solving energies with higher order cliques.
In: CVPR. (2007)
42.Baqu, P., Bagautdinov, T., Fleuret, F., Fua, P.: Principled Parallel Mean-Field Inference for
Discrete Random Fields. In: CVPR. (2016)
43.Hariharan, B., Arbel ´aez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse
detectors. In: ICCV , IEEE (2011) 991–998
44.Lin, T.Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll ´ar, P., Zitnick, C.L.:
Microsoft coco: Common objects in context. In: ECCV . Springer (2014) 740–755

ECCV 2016


Microsoft coco: Common objects in context. In: ECCV . Springer (2014) 740–755
45.Kokkinos, I.: Pushing the boundaries of boundary detection using deep learning. In: ICLR.
(2016)
46.Yu, F., Koltun, V .: Multi-scale context aggregation by dilated convolutions. In: ICLR. (2016)
47.Chen, L.C., Yang, Y ., Wang, J., Xu, W., Yuille, A.L.: Attention to scale: Scale-aware semantic
image segmentation. In: CVPR. (2016)
48.Papandreou, G., Chen, L., Murphy, K., Yuille, A.L.: Weakly- and semi-supervised learning of
a DCNN for semantic image segmentation. In: ICCV . (2015)
49.Liu, W., Rabinovich, A., Berg, A.C.: Parsenet: Looking wider to see better. arXiv preprint
arXiv:1506.04579 (2015)
Higher Order Conditional Random Fields in Deep Neural Networks 17
50.Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., et al.: The

ECCV 2016


role of context for object detection and semantic segmentation in the wild. In: CVPR, IEEE
(2014) 891–898
51.Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In:
ICCV . (2015)
52.Mostajabi, M., Yadollahpour, P., Shakhnarovich, G.: Feedforward semantic segmentation
with zoom-out features. In: CVPR. (2015)
53.Dong, J., Chen, Q., Yan, S., Yuille, A.: Towards uniﬁed object detection and semantic
segmentation. In: ECCV . (2014) 299–314
54.Carreira, J., Caseiro, R., Batista, J., Sminchisescu, C.: Free-form region description with
second-order pooling. PAMI (2014)
Appendix
Appendix A of this supplementary material presents the derivatives of the mean ﬁeld
updates which we use for inference in our Conditional Random Field (CRF). Appendix
B shows detailed qualitative results for the experiments described in our main paper.
A Derivatives of Mean Field Updates
The pseudocode for the mean ﬁeld inference algorithm with latent Ydetection variables

ECCV 2016


A Derivatives of Mean Field Updates
The pseudocode for the mean ﬁeld inference algorithm with latent Ydetection variables
is shown below in Algorithm 1. We use the same notation used in the main paper.
Algorithm 1 Mean Field Inference
Q0(Xi=l)←1
Ziexp/parenleftbig
−ψU
i(l)/parenrightbig
,∀i,l
Q0(Yd=b)←sb
d(1−sd)(1−b),∀d,b
⊿Initialisation
fort = 0 :T−1do
Et(Xi=l)←UnaryUpdate + PairwiseUpdate +
DetectionUpdate + SuperpixelUpdate, ∀i,l
Et(Yd=b)←YUnaryUpdate + Y DetectionUpdate
⊿Mean ﬁeld updates
Qt+1(Xi=l)←1
Ziexp/parenleftbig
−Et(Xi=l)/parenrightbig
,∀i,l
Qt+1(Yd=b)←1
Zdexp/parenleftbig
−Et(Yd=b)/parenrightbig
,∀d,b
⊿Normalising
end for

ECCV 2016


Zdexp/parenleftbig
−Et(Yd=b)/parenrightbig
,∀d,b
⊿Normalising
end for
For the explicit forms of the UnaryUpdate andPairwiseUpdate above, and their
differentials, we refer the reader to [10] and discuss the terms DetectionUpdate and
SuperpixelUpdate in detail below.
Let us assume that only one object detection of the form (ld,sd,Fd)is available for
the image under consideration. When multiple detections are present, simply a summation of the updates and differentials discussed below apply. Therefore, no generality
18 Arnab et al.
is lost with this assumption. Similarly, we can assume that only one superpixel clique
{Xs}is present, without a loss of generality.
Assuming that pixel iin Algorithm 1 belongs to Fd, Eq. (6) in the main paper
described the exact form of DetectionUpdate . Similarly, assuming that pixel ibelongs
to{Xs}Eq. (8) described the form of SuperpixelUpdate .
LetLdenote the value of the loss function calculated at the output of the deep
network. This could be the softmax loss or any other appropriate loss function. During
backpropagation, we get the error signal∂L

ECCV 2016


network. This could be the softmax loss or any other appropriate loss function. During
backpropagation, we get the error signal∂L
∂QTat the output of the mean ﬁeld inference.
Using this error information, we need to compute the derivative of the loss Lwith respect
to theXunaries and various CRF parameters. Note that, if we compute the relevent
differentials for only one iteration of the mean ﬁeld algorithm, it is possible to calculate
them for multiple iterations using the recurrent behaviour of the iterations.
Note that, by looking at Normalising step of Algorithm 1, it is trivial to calculate
∂Qt+1
∂Et. Therefore, we can then calculate∂L
∂Etusing the chain rule. This is same as
backpropagation of the usual softmax operation in a deep network (up to a negative sign).
Using this observation we can calculate the necessary differentials to take the forms
shown below:
∂L
∂wDet=sd
ndnd/summationdisplay
i=1/parenleftBig∂L
Et(X(i)
d=ld)Qt(Yd= 1) + (9)
/summationdisplay
l/prime/negationslash=ld∂L

ECCV 2016


d=ld)Qt(Yd= 1) + (9)
/summationdisplay
l/prime/negationslash=ld∂L
∂Et(X(i)
d=l/prime)Qt(Yd= 1)/parenrightBig
+
∂L
∂Et(Yd= 0)sd
ndnd/summationdisplay
i=1Qt(X(i)
d=ld) +
∂L
∂Et(Yd= 1)sd
ndnd/summationdisplay
i=1/parenleftBig
1−Qt(X(i)
d=ld)/parenrightBig
∂L
∂Qt(X(i)
d=ld)=wDet∂L
∂Et(Yd= 0)−wDet∂L
∂Et(Yd= 1)(10)
∂L
∂Qt(Yd= 0)=wDetsd
ndnd/summationdisplay
i=1/parenleftBigg
∂L
Et(X(i)
d=ld)/parenrightBigg
(11)
∂L
∂Qt(Yd= 1)=wDetsd
ndnd/summationdisplay

ECCV 2016


d=ld)/parenrightBigg
(11)
∂L
∂Qt(Yd= 1)=wDetsd
ndnd/summationdisplay
i=1/summationdisplay
l/negationslash=ld/parenleftBigg
∂L
∂Et(X(i)
d=l/prime)/parenrightBigg
(12)
∂L
∂wLow(l)=/summationdisplay
i∈s
∂L
∂Et(X(i)
s=l)/productdisplay
j∈c,j/negationslash=iQt(Xj=l)
 (13)
Higher Order Conditional Random Fields in Deep Neural Networks 19
∂L
∂wHigh=/summationdisplay
i∈s/summationdisplay
l∈L
∂L
∂Et(X(i)
s=l)
1−/productdisplay
j∈c,j/negationslash=iQt(Xj=l)

 (14)
Effect of the superpixel potentials on the derivatives∂L

ECCV 2016



 (14)
Effect of the superpixel potentials on the derivatives∂L
∂Qt(Xi=l)were negligible.
Therefore, we ignored them in our calculations.
B Additional Experimental Results
Table 5 presents more detailed results of our method, and that of other state-of-the-art
techniques, on the PASCAL VOC 2012 test set. In particular, we present the accuracy
for every class in the VOC test set. Note that our per-class accuracy improves over our
baseline, CRF-RNN [10], for all of the 20 classes in PASCAL VOC.
Figure 6 shows more sample results of our system, compared to our baseline, CRFas-RNN [10]. Figure 7 shows examples of failure cases of our method. Figure 8 examines
the effect of each of our potentials. Finally, Figure 9 shows a qualitative comparison
between the output of our system and other current methods on the PASCAL VOC 2012
test set.
20 Arnab et al.Table 5: Comparison of the mean Intersection over Union (IoU) accuracy of our approach and other state-of-the-art methods on the Pascal
VOC 2012 test set. Scores for other methods were taken from the original authors’ publications.
Methods trained with
COCOMean

ECCV 2016


ECCV 2016


O2P [54] 47.8 64.0 27.3 54.1 39.2 48.7 56.6 57.7 52.5 14.2 54.8 29.6 42.2 58.0 54.8 50.2 36.6 58.6 31.6 48.4 38.6
Higher Order Conditional Random Fields in Deep Neural Networks 21
Input image CRF-as-RNN [10] Our method Ground truth
Fig. 6: Examples of images where our method has improved over our baseline,
CRF-as-RNN [10]. The input images have the detection bounding boxes overlaid on
them. Note that the method of [10] does not make use of this information. The improvements from our method are due to our detection potentials, as well as our superpixel
based potentials. Note that all images are from the reduced validation set of VOC 2012
and have not been trained on at all. Best viewed in color.
22 Arnab et al.
Input image CRF-as-RNN [10] Our method Ground truth
Fig. 7: Examples of failure cases where our method has performed poorly. The ﬁrst
row shows an example of how the detection of the person has now resulted in the sofa
being misclassiﬁed (although our system is able to reject the other false detection). Our

ECCV 2016


row shows an example of how the detection of the person has now resulted in the sofa
being misclassiﬁed (although our system is able to reject the other false detection). Our
superpixel potentials have a tendency to remove spurious noise by enforcing consistency
within regions. However, as shown in the second row, sometimes the “noise” being
removed is actually the correct label. In the other cases, we are limited by our pixelwise
classiﬁcation unaries which are poor. Our superpixel and detection potentials are not
always able to compensate for this. Note that all images are from the reduced validation
set of VOC 2012 and have not been trained on at all. The input images have the detection
bounding boxes overlaid on them. Note that the method of [10] does not make use of
this information. Best viewed in color.
Higher Order Conditional Random Fields in Deep Neural Networks 23
Input image Pairwise onlySuperpixels
onlyDetections
onlyDetections
and
SuperpixelsGround truth
Fig. 8: Comparison of pairwise potentials, superpixel and pairwise potentials, detection and pairwise potentials, and a combination of all three (Row 1 and 2) These
are examples where superpixel potentials help to remove spurious noise in the output

ECCV 2016


are examples where superpixel potentials help to remove spurious noise in the output
but detection potentials do not affect the result. The ﬁnal result still improves when
all potentials are combined. (Row 3) Detection potentials greatly improve the result
by recognising the train correctly (the pixelwise unaries are largest for “bus”). And
superpixels, when combined with detections, slightly improve the output. (Row 4) An
example where both superpixel and detection potentials improve the ﬁnal output. (Row 5)
A case where the superpixel worsens the result as, although the output is more consistent
among superpixel regions, some pixels have had their correct labels removed. However,
the correct detection improves the result, and the output of combining superpixel and
detection potentials is actually better than either potential in isolation. (Row 6) Here, the
detection (although correct) worsens the output due to its imprecise foreground mask.
Superpixel potentials also exacerbate the result, since the legs of the chair and the chair’s
shadow are confused to be part of the same superpixel region. However, when the two
potentials are combined, the result is slightly better than with only detection potentials.
24 Arnab et al.

ECCV 2016


potentials are combined, the result is slightly better than with only detection potentials.
24 Arnab et al.
Input image FCN-8s [3] Deeplab [12] CRF-as-RNN [10] Our method Ground truth
Fig. 9: Qualitative comparison with other current methods. Sample results of our
method compared to other current techniques on VOC 2012. We reproduced the segmentation results of Deeplab from their original publication, whilst we reproduced the results
of FCN-8s and CRF-as-RNN from their publicly-available source code. Best viewed in
colour.

ECCV 2016


Deep Speech 2: End-to-End Speech Recognition in
English and Mandarin
Baidu Research – Silicon Valley AI Lab
Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro,
Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel,
Linxi Fan, Christopher Fougner, Tony Han, Awni Hannun, Billy Jun, Patrick LeGresley,
Libby Lin, Sharan Narang, Andrew Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman,
Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang,
Bo Xiao, Dani Yogatama, Jun Zhan, Zhenyao Zhu
Abstract
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech—two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is
our application of HPC techniques, resulting in a 7x speedup over our previous

Summary:
We show that an end-to-end deep learning approach can be used to recognize
either English or Mandarin Chinese speech--two vastly different languages.
Because it replaces entire pipelines of hand-engineered components with neural
networks, end-to-end learning allows us to handle a diverse variety of speech
including noisy environments, accents and different languages. Key to our
approach is our application of HPC techniques, resulting in a 7x speedup over
our previous system. Because of this efficiency, experiments that previously
took weeks now run in days. This enables us to iterate more quickly to identify
superior architectures and algorithms. As a result, in several cases, our
system is competitive with the transcription of human workers when benchmarked
on standard datasets. Finally, using a technique called Batch Dispatch with
GPUs in the data center, we show that our system can be inexpensively deployed
in an online setting, delivering low latency when serving users at scale.

Dario Amodei
Rishita Anubhai
Eric Battenberg
Carl Case
Jared Casper
Bryan Catanzaro
Jingdong Chen
Mike Chrzanowski
Adam Coates
Greg Diamos
Erich Elsen
Jesse Engel
Linxi Fan
Christopher Fougner
Tony Han
Awni Hannun
Billy Jun
Patrick LeGresley
Libby Lin
Sharan Narang
Andrew Ng
Sherjil Ozair
Ryan Prenger
Jonathan Raiman
Sanjeev Satheesh
David Seetapun
Shubho Sengupta
Yi Wang
Zhiqian Wang
Chong Wang
Bo Xiao
Dani Yogatama
Jun Zhan
Zhenyao Zhu


our application of HPC techniques, resulting in a 7x speedup over our previous
system [26]. Because of this efﬁciency, experiments that previously took weeks
now run in days. This enables us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive
with the transcription of human workers when benchmarked on standard datasets.
Finally, using a technique called Batch Dispatch with GPUs in the data center, we
show that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.
1 Introduction
Decades worth of hand-engineered domain knowledge has gone into current state-of-the-art automatic speech recognition (ASR) pipelines. A simple but powerful alternative solution is to train such
ASR models end-to-end, using deep learning to replace most modules with a single model [26]. We
present the second generation of our speech system that exempliﬁes the major advantages of endto-end learning. The Deep Speech 2 ASR pipeline approaches or exceeds the accuracy of Amazon
Mechanical Turk human workers on several benchmarks, works in multiple languages with little
modiﬁcation, and is deployable in a production setting. It thus represents a signiﬁcant step towards


modiﬁcation, and is deployable in a production setting. It thus represents a signiﬁcant step towards
a single ASR system that addresses the entire range of speech recognition contexts handled by humans. Since our system is built on end-to-end deep learning, we can employ a spectrum of deep
learning techniques: capturing large training sets, training larger models with high performance
computing, and methodically exploring the space of neural network architectures. We show that
through these techniques we are able to reduce error rates of our previous end-to-end system [26] in
English by up to 43%, and can also recognize Mandarin speech with high accuracy.
One of the challenges of speech recognition is the wide range of variability in speech and acoustics.
As a result, modern ASR pipelines are made up of numerous components including complex feature
extraction, acoustic models, language and pronunciation models, speaker adaptation, etc. Building and tuning these individual components makes developing a new speech recognizer very hard,
especially for a new language. Indeed, many parts do not generalize well across environments or
languages and it is often necessary to support multiple application-speciﬁc systems in order to provide acceptable accuracy. This state of affairs is different from human speech recognition: people
Authorship order is alphabetical.