berkeleyvision.org/ , 2013.
Koller, D. and Friedman, N. Probabilistic Graphical Models: Principles and Techniques . MIT Press, 2009.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet Classiﬁcation with Deep Convolutional Neural
Networks. In Proc. NIPS , 2013.
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., and Bengio, Y . An empirical evaluation of deep architectures on problems with many factors of variation. In Proc. ICML , 2007.
LeCun, Y ., Bottou, L., Bengio, Y ., and Haffner, P. Gradient-based learning applied to document recognition.
Proceedings of the IEEE , 1998.
Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y . Convolutional deep belief networks for scalable unsupervised
learning of hierarchical representations. In Proc. ICML , 2009.
Li, Y . and Zemel, R. High Order Regularization for Semi-Supervised Learning of Structured Output Problems.
InProc. ICML , 2014.
10
Accepted as a workshop contribution at ICLR 2015

Learning Deep Structured Models

11 pages including reference


InProc. ICML , 2014.
10
Accepted as a workshop contribution at ICLR 2015
Ma, J., Peng, J., Wang, S., and Xu, J. A conditional neural ﬁelds model for protein threading. Bioinformatics ,
2012.
Meltzer, T., Globerson, A., and Weiss, Y . Convergent Message Passing Algorithms: a unifying view. In Proc.
UAI, 2009.
Meshi, O., Sontag, D., Jaakkola, T., and Globerson, A. Learning Efﬁciently with Approximate inference via
Dual Losses. In Proc. ICML , 2010.
Morris, J. and Fosler-Lussier, E. Conditional random ﬁelds for integrating local discriminative classiﬁers. IEEE
Trans. Audio, Speech, and Language Processing , 2008.
Nowozin, S., Rother, C., Bagon, S., Sharp, T., Yao, B., and Kohli, P. Decision tree ﬁelds. In Proc. ICCV , 2011.
Pearl, J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgan Kaufmann,
1988.

Learning Deep Structured Models

Summary:
Many problems in real-world applications involve predicting several random
variables which are statistically related. Markov random fields (MRFs) are a
great mathematical tool to encode such relationships. The goal of this paper is
to combine MRFs with deep learning algorithms to estimate complex
representations while taking into account the dependencies between the output
random variables. Towards this goal, we propose a training algorithm that is
able to learn structured models jointly with deep features that form the MRF
potentials. Our approach is efficient as it blends learning and inference and
makes use of GPU acceleration. We demonstrate the effectiveness of our
algorithm in the tasks of predicting words from noisy images, as well as
multi-class classification of Flickr photographs. We show that joint learning
of the deep features and the MRF parameters results in significant performance
gains.

Liang-Chieh Chen
Alexander G. Schwing
Alan L. Yuille
Raquel Urtasun

11 pages including reference


Pearl, J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference . Morgan Kaufmann,
1988.
Peng, J., Bo, L., and Xu, J. Conditional Neural Fields. In Proc. NIPS , 2009.
Prabhavalkar, R. and Fosler-Lussier, E. Backpropagation training for multilayer conditional random ﬁeld based
phone recognition. In Proc. ICASSP , 2010.
Salakhutdinov, R. R. and Hinton, G. E. An Efﬁcient Learning Procedure for Deep Boltzmann Machines. Neural
Computation , 2012.
Schwing, A. G. Inference and Learning Algorithms with Applications to 3D Indoor Scene Understanding . PhD
thesis, ETH Zurich, 2013.
Socher, R., Huval, B., Bhat, B., Manning, C. D., and Ng, A. Y . Convolutional-Recursive Deep Learning for 3D
Object Classiﬁcation. In Proc. NIPS , 2012.
Tompson, J., Jain, A., LeCun, Y ., and Bregler, C. Joint Training of a Convolutional Network and a Graphical
Model for Human Pose Estimation. In Proc. NIPS , 2014.

Learning Deep Structured Models

11 pages including reference


Model for Human Pose Estimation. In Proc. NIPS , 2014.
Wainwright, M. J. and Jordan, M. I. Graphical Models, Exponential Families and Variational Inference . Foundations and Trends in Machine Learning, 2008.
Wainwright, M. J., Jaakkola, T., and Willsky, A. S. A new class of upper bounds on the log partition function.
Trans. Information Theory , 2005.
Weiss, Y ., Yanover, C., and Meltzer, T. MAP Estimation, Linear Programming and Belief Propagation with
Convex Free Energies. In Proc. UAI , 2007.
Wiegerinck, W. and Heskes, T. Fractional belief propagation. In Proc. NIPS , 2003.
Xu, J., Schwing, A. G., and Urtasun, R. Tell me what you see and I will show you where it is. In Proc. CVPR ,
2014.
Yedidia, J. S., Freeman, W. T., and Weiss, Y . Constructing free-energy approximations and generalized belief
propagation algorithms. Trans. Information Theory , 2005.
Zeiler, M. D. and Fergus, R. Visualizing and Understanding Convolutional Networks. In Proc. ECCV , 2014.
11

Learning Deep Structured Models

11 pages including reference


Published as a conference paper at ICLR 2015
NEURAL MACHINE TRANSLATION
BYJOINTLY LEARNING TO ALIGN AND TRANSLATE
Dzmitry Bahdanau
Jacobs University Bremen, Germany
KyungHyun Cho Yoshua Bengio
Universit ´e de Montr ´eal
ABSTRACT
Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine
translation aims at building a single neural network that can be jointly tuned to
maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder–decoders and encode
a source sentence into a ﬁxed-length vector from which a decoder generates a
translation. In this paper, we conjecture that the use of a ﬁxed-length vector is a
bottleneck in improving the performance of this basic encoder–decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search
for parts of a source sentence that are relevant to predicting a target word, without
having to form these parts as a hard segment explicitly. With this new approach,
we achieve a translation performance comparable to the existing state-of-the-art
phrase-based system on the task of English-to-French translation. Furthermore,

Neural Machine Translation by Jointly Learning to Align and Translate

Summary:
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.

Dzmitry Bahdanau
Kyunghyun Cho
Yoshua Bengio

Accepted at ICLR 2015 as oral presentation


we achieve a translation performance comparable to the existing state-of-the-art
phrase-based system on the task of English-to-French translation. Furthermore,
qualitative analysis reveals that the (soft-)alignments found by the model agree
well with our intuition.
1 I NTRODUCTION
Neural machine translation is a newly emerging approach to machine translation, recently proposed
by Kalchbrenner and Blunsom (2013), Sutskever et al. (2014) and Cho et al. (2014b). Unlike the
traditional phrase-based translation system (see, e.g., Koehn et al. , 2003) which consists of many
small sub-components that are tuned separately, neural machine translation attempts to build and
train a single, large neural network that reads a sentence and outputs a correct translation.
Most of the proposed neural machine translation models belong to a family of encoder–
decoders (Sutskever et al. , 2014; Cho et al. , 2014a), with an encoder and a decoder for each language, or involve a language-speciﬁc encoder applied to each sentence whose outputs are then compared (Hermann and Blunsom, 2014). An encoder neural network reads and encodes a source sentence into a ﬁxed-length vector. A decoder then outputs a translation from the encoded vector. The


whole encoder–decoder system, which consists of the encoder and the decoder for a language pair,
is jointly trained to maximize the probability of a correct translation given a source sentence.
A potential issue with this encoder–decoder approach is that a neural network needs to be able to
compress all the necessary information of a source sentence into a ﬁxed-length vector. This may
make it difﬁcult for the neural network to cope with long sentences, especially those that are longer
than the sentences in the training corpus. Cho et al. (2014b) showed that indeed the performance of
a basic encoder–decoder deteriorates rapidly as the length of an input sentence increases.
In order to address this issue, we introduce an extension to the encoder–decoder model which learns
to align and translate jointly. Each time the proposed model generates a word in a translation, it
(soft-)searches for a set of positions in a source sentence where the most relevant information is
concentrated. The model then predicts a target word based on the context vectors associated with
these source positions and all the previous generated target words.
CIFAR Senior Fellow
1arXiv:1409.0473v7  [cs.CL]  19 May 2016
Published as a conference paper at ICLR 2015


CIFAR Senior Fellow
1arXiv:1409.0473v7  [cs.CL]  19 May 2016
Published as a conference paper at ICLR 2015
The most important distinguishing feature of this approach from the basic encoder–decoder is that
it does not attempt to encode a whole input sentence into a single ﬁxed-length vector. Instead, it encodes the input sentence into a sequence of vectors and chooses a subset of these vectors adaptively
while decoding the translation. This frees a neural translation model from having to squash all the
information of a source sentence, regardless of its length, into a ﬁxed-length vector. We show this
allows a model to cope better with long sentences.
In this paper, we show that the proposed approach of jointly learning to align and translate achieves
signiﬁcantly improved translation performance over the basic encoder–decoder approach. The improvement is more apparent with longer sentences, but can be observed with sentences of any
length. On the task of English-to-French translation, the proposed approach achieves, with a single
model, a translation performance comparable, or close, to the conventional phrase-based system.
Furthermore, qualitative analysis reveals that the proposed model ﬁnds a linguistically plausible
(soft-)alignment between a source sentence and the corresponding target sentence.


Furthermore, qualitative analysis reveals that the proposed model ﬁnds a linguistically plausible
(soft-)alignment between a source sentence and the corresponding target sentence.
2 B ACKGROUND : NEURAL MACHINE TRANSLATION
From a probabilistic perspective, translation is equivalent to ﬁnding a target sentence ythat maximizes the conditional probability of ygiven a source sentence x, i.e., arg maxyp(yjx). In
neural machine translation, we ﬁt a parameterized model to maximize the conditional probability
of sentence pairs using a parallel training corpus. Once the conditional distribution is learned by a
translation model, given a source sentence a corresponding translation can be generated by searching
for the sentence that maximizes the conditional probability.
Recently, a number of papers have proposed the use of neural networks to directly learn this conditional distribution (see, e.g., Kalchbrenner and Blunsom, 2013; Cho et al. , 2014a; Sutskever et al. ,
2014; Cho et al. , 2014b; Forcada and ˜Neco, 1997). This neural machine translation approach typically consists of two components, the ﬁrst of which encodes a source sentence xand the second
decodes to a target sentence y. For instance, two recurrent neural networks (RNN) were used by


decodes to a target sentence y. For instance, two recurrent neural networks (RNN) were used by
(Cho et al. , 2014a) and (Sutskever et al. , 2014) to encode a variable-length source sentence into a
ﬁxed-length vector and to decode the vector into a variable-length target sentence.
Despite being a quite new approach, neural machine translation has already shown promising results.
Sutskever et al. (2014) reported that the neural machine translation based on RNNs with long shortterm memory (LSTM) units achieves close to the state-of-the-art performance of the conventional
phrase-based machine translation system on an English-to-French translation task.1Adding neural
components to existing translation systems, for instance, to score the phrase pairs in the phrase
table (Cho et al. , 2014a) or to re-rank candidate translations (Sutskever et al. , 2014), has allowed to
surpass the previous state-of-the-art performance level.
2.1 RNN E NCODER –DECODER
Here, we describe brieﬂy the underlying framework, called RNN Encoder–Decoder , proposed by
Cho et al. (2014a) and Sutskever et al. (2014) upon which we build a novel architecture that learns


Cho et al. (2014a) and Sutskever et al. (2014) upon which we build a novel architecture that learns
to align and translate simultaneously.
In the Encoder–Decoder framework, an encoder reads the input sentence, a sequence of vectors
x= (x1;;xTx), into a vector c.2The most common approach is to use an RNN such that
ht=f(xt;ht1) (1)
and
c=q(fh1;;hTxg);
whereht2Rnis a hidden state at time t, andcis a vector generated from the sequence of the
hidden states. fandqare some nonlinear functions. Sutskever et al. (2014) used an LSTM as fand
q(fh1;;hTg) =hT, for instance.
1We mean by the state-of-the-art performance, the performance of the conventional phrase-based system
without using any neural network-based component.
2Although most of the previous works (see, e.g., Cho et al. , 2014a; Sutskever et al. , 2014; Kalchbrenner and


2Although most of the previous works (see, e.g., Cho et al. , 2014a; Sutskever et al. , 2014; Kalchbrenner and
Blunsom, 2013) used to encode a variable-length input sentence into a ﬁxed-length vector, it is not necessary,
and even it may be beneﬁcial to have a variable-length vector, as we will show later.
2
Published as a conference paper at ICLR 2015
The decoder is often trained to predict the next word yt0given the context vector cand all the
previously predicted words fy1;;yt01g. In other words, the decoder deﬁnes a probability over
the translation yby decomposing the joint probability into the ordered conditionals:
p(y) =TY
t=1p(ytjfy1;;yt1g;c); (2)
where y=
y1;;yTy
. With an RNN, each conditional probability is modeled as
p(ytjfy1;;yt1g;c) =g(yt1;st;c); (3)
wheregis a nonlinear, potentially multi-layered, function that outputs the probability of yt, andstis


wheregis a nonlinear, potentially multi-layered, function that outputs the probability of yt, andstis
the hidden state of the RNN. It should be noted that other architectures such as a hybrid of an RNN
and a de-convolutional neural network can be used (Kalchbrenner and Blunsom, 2013).
3 L EARNING TO ALIGN AND TRANSLATE
In this section, we propose a novel architecture for neural machine translation. The new architecture
consists of a bidirectional RNN as an encoder (Sec. 3.2) and a decoder that emulates searching
through a source sentence during decoding a translation (Sec. 3.1).
3.1 D ECODER : GENERAL DESCRIPTION
x1x2x3xT+
αt,1
αt,2 αt,3αt,Tyt-1yt
h1h2h3 hTh1h2h3 hTst-1st
Figure 1: The graphical illustration of the proposed model
trying to generate the t-th target wordytgiven a source
sentence (x1;x2;:::;x T).In a new model architecture, we deﬁne each conditional probability
in Eq. (2) as:


sentence (x1;x2;:::;x T).In a new model architecture, we deﬁne each conditional probability
in Eq. (2) as:
p(yijy1;:::;y i1;x) =g(yi1;si;ci); (4)
wheresiis an RNN hidden state for time i, computed by
si=f(si1;yi1;ci):
It should be noted that unlike the existing encoder–decoder approach (see Eq. (2)), here the probability is conditioned on a distinct
context vector cifor each target word yi.
The context vector cidepends on a sequence of annotations
(h1;;hTx)to which an encoder maps the input sentence. Each
annotationhicontains information about the whole input sequence
with a strong focus on the parts surrounding the i-th word of the
input sequence. We explain in detail how the annotations are computed in the next section.
The context vector ciis, then, computed as a weighted sum of these
annotationshi:
ci=TxX
j=1ijhj: (5)
The weightijof each annotation hjis computed by
ij=exp (eij)PTx


ci=TxX
j=1ijhj: (5)
The weightijof each annotation hjis computed by
ij=exp (eij)PTx
k=1exp (eik); (6)
where
eij=a(si1;hj)
is an alignment model which scores how well the inputs around position jand the output at position
imatch. The score is based on the RNN hidden state si1(just before emitting yi, Eq. (4)) and the
j-th annotation hjof the input sentence.
We parametrize the alignment model aas a feedforward neural network which is jointly trained with
all the other components of the proposed system. Note that unlike in traditional machine translation,
3
Published as a conference paper at ICLR 2015
the alignment is not considered to be a latent variable. Instead, the alignment model directly computes a soft alignment, which allows the gradient of the cost function to be backpropagated through.
This gradient can be used to train the alignment model as well as the whole translation model jointly.
We can understand the approach of taking a weighted sum of all the annotations as computing an
expected annotation , where the expectation is over possible alignments. Let ijbe a probability that


We can understand the approach of taking a weighted sum of all the annotations as computing an
expected annotation , where the expectation is over possible alignments. Let ijbe a probability that
the target word yiis aligned to, or translated from, a source word xj. Then, thei-th context vector
ciis the expected annotation over all the annotations with probabilities ij.
The probability ij, or its associated energy eij, reﬂects the importance of the annotation hjwith
respect to the previous hidden state si1in deciding the next state siand generating yi. Intuitively,
this implements a mechanism of attention in the decoder. The decoder decides parts of the source
sentence to pay attention to. By letting the decoder have an attention mechanism, we relieve the
encoder from the burden of having to encode all information in the source sentence into a ﬁxedlength vector. With this new approach the information can be spread throughout the sequence of
annotations, which can be selectively retrieved by the decoder accordingly.
3.2 E NCODER : BIDIRECTIONAL RNN FOR ANNOTATING SEQUENCES
The usual RNN, described in Eq. (1), reads an input sequence xin order starting from the ﬁrst


The usual RNN, described in Eq. (1), reads an input sequence xin order starting from the ﬁrst
symbolx1to the last one xTx. However, in the proposed scheme, we would like the annotation
of each word to summarize not only the preceding words, but also the following words. Hence,
we propose to use a bidirectional RNN (BiRNN, Schuster and Paliwal, 1997), which has been
successfully used recently in speech recognition (see, e.g., Graves et al. , 2013).
A BiRNN consists of forward and backward RNN’s. The forward RNN !freads the input sequence
as it is ordered (from x1toxTx) and calculates a sequence of forward hidden states ( !h1;; !hTx).
The backward RNN freads the sequence in the reverse order (from xTxtox1), resulting in a
sequence of backward hidden states ( h1;; hTx).
We obtain an annotation for each word xjby concatenating the forward hidden state !hjand the
backward one hj, i.e.,hj=h !h>
j; h>
ji>


backward one hj, i.e.,hj=h !h>
j; h>
ji>
. In this way, the annotation hjcontains the summaries
of both the preceding words and the following words. Due to the tendency of RNNs to better
represent recent inputs, the annotation hjwill be focused on the words around xj. This sequence
of annotations is used by the decoder and the alignment model later to compute the context vector
(Eqs. (5)–(6)).
See Fig. 1 for the graphical illustration of the proposed model.
4 E XPERIMENT SETTINGS
We evaluate the proposed approach on the task of English-to-French translation. We use the bilingual, parallel corpora provided by ACL WMT ’14.3As a comparison, we also report the performance of an RNN Encoder–Decoder which was proposed recently by Cho et al. (2014a). We use
the same training procedures and the same dataset for both models.4
4.1 D ATASET
WMT ’14 contains the following English-French parallel corpora: Europarl (61M words), news
commentary (5.5M), UN (421M) and two crawled corpora of 90M and 272.5M words respectively,


commentary (5.5M), UN (421M) and two crawled corpora of 90M and 272.5M words respectively,
totaling 850M words. Following the procedure described in Cho et al. (2014a), we reduce the size of
the combined corpus to have 348M words using the data selection method by Axelrod et al. (2011).5
We do not use any monolingual data other than the mentioned parallel corpora, although it may be
possible to use a much larger monolingual corpus to pretrain an encoder. We concatenate news-test3http://www.statmt.org/wmt14/translation-task.html
4Implementations are available at https://github.com/lisa-groundhog/GroundHog .
5Available online at http://www-lium.univ-lemans.fr/ ˜schwenk/cslm_joint_paper/ .
4
Published as a conference paper at ICLR 2015
0 10 20 30 40 50 60
Sentence length051015202530BLEU score RNNsearch-50
RNNsearch-30
RNNenc-50
RNNenc-30
Figure 2: The BLEU scores
of the generated translations
on the test set with respect
to the lengths of the sentences. The results are on


RNNenc-30
Figure 2: The BLEU scores
of the generated translations
on the test set with respect
to the lengths of the sentences. The results are on
the full test set which includes sentences having unknown words to the models.
2012 and news-test-2013 to make a development (validation) set, and evaluate the models on the test
set (news-test-2014) from WMT ’14, which consists of 3003 sentences not present in the training
data.
After a usual tokenization6, we use a shortlist of 30,000 most frequent words in each language to
train our models. Any word not included in the shortlist is mapped to a special token ( [UNK ]). We
do not apply any other special preprocessing, such as lowercasing or stemming, to the data.
4.2 M ODELS
We train two types of models. The ﬁrst one is an RNN Encoder–Decoder (RNNencdec, Cho et al. ,
2014a), and the other is the proposed model, to which we refer as RNNsearch. We train each model
twice: ﬁrst with the sentences of length up to 30 words (RNNencdec-30, RNNsearch-30) and then


twice: ﬁrst with the sentences of length up to 30 words (RNNencdec-30, RNNsearch-30) and then
with the sentences of length up to 50 word (RNNencdec-50, RNNsearch-50).
The encoder and decoder of the RNNencdec have 1000 hidden units each.7The encoder of the
RNNsearch consists of forward and backward recurrent neural networks (RNN) each having 1000
hidden units. Its decoder has 1000 hidden units. In both cases, we use a multilayer network with a
single maxout (Goodfellow et al. , 2013) hidden layer to compute the conditional probability of each
target word (Pascanu et al. , 2014).
We use a minibatch stochastic gradient descent (SGD) algorithm together with Adadelta (Zeiler,
2012) to train each model. Each SGD update direction is computed using a minibatch of 80 sentences. We trained each model for approximately 5 days.
Once a model is trained, we use a beam search to ﬁnd a translation that approximately maximizes the
conditional probability (see, e.g., Graves, 2012; Boulanger-Lewandowski et al. , 2013). Sutskever
et al. (2014) used this approach to generate translations from their neural machine translation model.


et al. (2014) used this approach to generate translations from their neural machine translation model.
For more details on the architectures of the models and training procedure used in the experiments,
see Appendices A and B.
5 R ESULTS
5.1 Q UANTITATIVE RESULTS
In Table 1, we list the translation performances measured in BLEU score. It is clear from the table
that in all the cases, the proposed RNNsearch outperforms the conventional RNNencdec. More
importantly, the performance of the RNNsearch is as high as that of the conventional phrase-based
translation system (Moses), when only the sentences consisting of known words are considered.
This is a signiﬁcant achievement, considering that Moses uses a separate monolingual corpus (418M
words) in addition to the parallel corpora we used to train the RNNsearch and RNNencdec.
6We used the tokenization script from the open-source machine translation package, Moses.
7In this paper, by a ’hidden unit’, we always mean the gated hidden unit (see Appendix A.1.1).
5
Published as a conference paper at ICLR 2015
The
agreement
on
the
European
Economic
Area
was
signed
in
August
1992
.
<end>
L'
accord


The
agreement
on
the
European
Economic
Area
was
signed
in
August
1992
.
<end>
L'
accord
sur
la
zone
économique
européenne
a
été
signé
en
août
1992
.
<end>
It
should
be
noted
that
the
marine
environment
is
the
least
known
of
environments
.
<end>
Il
convient
de
noter
que
l'
environnement
marin
est
le
moins
connu
de
l'
environnement
.
<end>
(a) (b)
Destruction
of
the
equipment
means
that
Syria
can
no
longer
produce
new
chemical
weapons
.
<end>
La
destruction
de
l'
équipement
signifie
que
la
Syrie
ne
peut
plus
produire
de
nouvelles
armes
chimiques
.
<end>
"
This
will
change
my
future
with
my
family
,
"
the
man
said
.
<end>
"
Cela


"
This
will
change
my
future
with
my
family
,
"
the
man
said
.
<end>
"
Cela
va
changer
mon
avenir
avec
ma
famille
"
,
a
dit
l'
homme
.
<end>
(c) (d)
Figure 3: Four sample alignments found by RNNsearch-50. The x-axis and y-axis of each plot
correspond to the words in the source sentence (English) and the generated translation (French),
respectively. Each pixel shows the weight ijof the annotation of the j-th source word for the i-th
target word (see Eq. (6)), in grayscale ( 0: black, 1: white). (a) an arbitrary sentence. (b–d) three
randomly selected samples among the sentences without any unknown words and of length between
10 and 20 words from the test set.
One of the motivations behind the proposed approach was the use of a ﬁxed-length context vector
in the basic encoder–decoder approach. We conjectured that this limitation may make the basic


in the basic encoder–decoder approach. We conjectured that this limitation may make the basic
encoder–decoder approach to underperform with long sentences. In Fig. 2, we see that the performance of RNNencdec dramatically drops as the length of the sentences increases. On the other hand,
both RNNsearch-30 and RNNsearch-50 are more robust to the length of the sentences. RNNsearch50, especially, shows no performance deterioration even with sentences of length 50 or more. This
superiority of the proposed model over the basic encoder–decoder is further conﬁrmed by the fact
that the RNNsearch-30 even outperforms RNNencdec-50 (see Table 1).
6
Published as a conference paper at ICLR 2015
Model All No UNK
RNNencdec-30 13.93 24.19
RNNsearch-30 21.50 31.44
RNNencdec-50 17.82 26.71
RNNsearch-50 26.75 34.16
RNNsearch-50?28.45 36.15
Moses 33.30 35.63Table 1: BLEU scores of the trained models computed on the test set. The second and third columns
show respectively the scores on all the sentences and,


show respectively the scores on all the sentences and,
on the sentences without any unknown word in themselves and in the reference translations. Note that
RNNsearch-50?was trained much longer until the
performance on the development set stopped improving. () We disallowed the models to generate [UNK]
tokens when only the sentences having no unknown
words were evaluated (last column).
5.2 Q UALITATIVE ANALYSIS
5.2.1 A LIGNMENT
The proposed approach provides an intuitive way to inspect the (soft-)alignment between the words
in a generated translation and those in a source sentence. This is done by visualizing the annotation
weightsijfrom Eq. (6), as in Fig. 3. Each row of a matrix in each plot indicates the weights
associated with the annotations. From this we see which positions in the source sentence were
considered more important when generating the target word.
We can see from the alignments in Fig. 3 that the alignment of words between English and French
is largely monotonic. We see strong weights along the diagonal of each matrix. However, we also
observe a number of non-trivial, non-monotonic alignments. Adjectives and nouns are typically
ordered differently between French and English, and we see an example in Fig. 3 (a). From this


ordered differently between French and English, and we see an example in Fig. 3 (a). From this
ﬁgure, we see that the model correctly translates a phrase [European Economic Area] into [zone
´economique europ ´een]. The RNNsearch was able to correctly align [zone] with [Area], jumping
over the two words ([European] and [Economic]), and then looked one word back at a time to
complete the whole phrase [zone ´economique europ ´eenne].
The strength of the soft-alignment, opposed to a hard-alignment, is evident, for instance, from
Fig. 3 (d). Consider the source phrase [the man] which was translated into [l’ homme]. Any hard
alignment will map [the] to [l’] and [man] to [homme]. This is not helpful for translation, as one
must consider the word following [the] to determine whether it should be translated into [le], [la],
[les] or [l’]. Our soft-alignment solves this issue naturally by letting the model look at both [the] and
[man], and in this example, we see that the model was able to correctly translate [the] into [l’]. We


[man], and in this example, we see that the model was able to correctly translate [the] into [l’]. We
observe similar behaviors in all the presented cases in Fig. 3. An additional beneﬁt of the soft alignment is that it naturally deals with source and target phrases of different lengths, without requiring a
counter-intuitive way of mapping some words to or from nowhere ([NULL]) (see, e.g., Chapters 4
and 5 of Koehn, 2010).
5.2.2 L ONG SENTENCES
As clearly visible from Fig. 2 the proposed model (RNNsearch) is much better than the conventional
model (RNNencdec) at translating long sentences. This is likely due to the fact that the RNNsearch
does not require encoding a long sentence into a ﬁxed-length vector perfectly, but only accurately
encoding the parts of the input sentence that surround a particular word.
As an example, consider this source sentence from the test set:
An admitting privilege is the right of a doctor to admit a patient to a hospital or
a medical centre tocarry outadiagnosis oraprocedure, based onhisstatus asa
health care worker atahospital.
The RNNencdec-50 translated this sentence into:


health care worker atahospital.
The RNNencdec-50 translated this sentence into:
Un privil `ege d’admission est le droit d’un m ´edecin de reconna ˆıtre un patient `a
l’hˆopital ou un centre m ´edical d’un diagnostic oudeprendre undiagnostic en
fonction deson´etatdesant´e.
7
Published as a conference paper at ICLR 2015
The RNNencdec-50 correctly translated the source sentence until [a medical center]. However, from
there on (underlined), it deviated from the original meaning of the source sentence. For instance, it
replaced [based on his status as a health care worker at a hospital] in the source sentence with [en
fonction de son ´etat de sant ´e] (“based on his state of health”).
On the other hand, the RNNsearch-50 generated the following correct translation, preserving the
whole meaning of the input sentence without omitting any details:
Un privil `ege d’admission est le droit d’un m ´edecin d’admettre un patient `a un


Un privil `ege d’admission est le droit d’un m ´edecin d’admettre un patient `a un
hˆopital ou un centre m ´edical pour effectuer undiagnostic ouuneproc´edure, selon
sonstatut detravailleur dessoins desant´e`al’hˆopital.
Let us consider another sentence from the test set:
This kind of experience is part of Disney’s efforts to ”extend the lifetime of its
series and build new relationships with audiences viadigital platforms that are
becoming ever more important, ” headded.
The translation by the RNNencdec-50 is
Ce type d’exp ´erience fait partie des initiatives du Disney pour ”prolonger la dur ´ee
de vie de ses nouvelles et de d ´evelopper des liens avec les lecteurs num´eriques qui
deviennent plus complexes.
As with the previous example, the RNNencdec began deviating from the actual meaning of the
source sentence after generating approximately 30 words (see the underlined phrase). After that


As with the previous example, the RNNencdec began deviating from the actual meaning of the
source sentence after generating approximately 30 words (see the underlined phrase). After that
point, the quality of the translation deteriorates, with basic mistakes such as the lack of a closing
quotation mark.
Again, the RNNsearch-50 was able to translate this long sentence correctly:
Ce genre d’exp ´erience fait partie des efforts de Disney pour ”prolonger la dur ´ee
de vie de ses s ´eries et cr ´eer de nouvelles relations avec des publics viades
plateformes num´eriques deplus enplus importantes”, a-t-il ajout ´e.
In conjunction with the quantitative results presented already, these qualitative observations conﬁrm our hypotheses that the RNNsearch architecture enables far more reliable translation of long
sentences than the standard RNNencdec model.
In Appendix C, we provide a few more sample translations of long source sentences generated by
the RNNencdec-50, RNNsearch-50 and Google Translate along with the reference translations.
6 R ELATED WORK
6.1 L EARNING TO ALIGN
A similar approach of aligning an output symbol with an input symbol was proposed recently by


6 R ELATED WORK
6.1 L EARNING TO ALIGN
A similar approach of aligning an output symbol with an input symbol was proposed recently by
Graves (2013) in the context of handwriting synthesis. Handwriting synthesis is a task where the
model is asked to generate handwriting of a given sequence of characters. In his work, he used a
mixture of Gaussian kernels to compute the weights of the annotations, where the location, width
and mixture coefﬁcient of each kernel was predicted from an alignment model. More speciﬁcally,
his alignment was restricted to predict the location such that the location increases monotonically.
The main difference from our approach is that, in (Graves, 2013), the modes of the weights of the
annotations only move in one direction. In the context of machine translation, this is a severe limitation, as (long-distance) reordering is often needed to generate a grammatically correct translation
(for instance, English-to-German).
Our approach, on the other hand, requires computing the annotation weight of every word in the
source sentence for each word in the translation. This drawback is not severe with the task of
translation in which most of input and output sentences are only 15–40 words. However, this may
limit the applicability of the proposed scheme to other tasks.
8


translation in which most of input and output sentences are only 15–40 words. However, this may
limit the applicability of the proposed scheme to other tasks.
8
Published as a conference paper at ICLR 2015
6.2 N EURAL NETWORKS FOR MACHINE TRANSLATION
Since Bengio et al. (2003) introduced a neural probabilistic language model which uses a neural network to model the conditional probability of a word given a ﬁxed number of the preceding words,
neural networks have widely been used in machine translation. However, the role of neural networks has been largely limited to simply providing a single feature to an existing statistical machine
translation system or to re-rank a list of candidate translations provided by an existing system.
For instance, Schwenk (2012) proposed using a feedforward neural network to compute the score of
a pair of source and target phrases and to use the score as an additional feature in the phrase-based
statistical machine translation system. More recently, Kalchbrenner and Blunsom (2013) and Devlin
et al. (2014) reported the successful use of the neural networks as a sub-component of the existing
translation system. Traditionally, a neural network trained as a target-side language model has been
used to rescore or rerank a list of candidate translations (see, e.g., Schwenk et al. , 2006).


used to rescore or rerank a list of candidate translations (see, e.g., Schwenk et al. , 2006).
Although the above approaches were shown to improve the translation performance over the stateof-the-art machine translation systems, we are more interested in a more ambitious objective of
designing a completely new translation system based on neural networks. The neural machine translation approach we consider in this paper is therefore a radical departure from these earlier works.
Rather than using a neural network as a part of the existing system, our model works on its own and
generates a translation from a source sentence directly.
7 C ONCLUSION
The conventional approach to neural machine translation, called an encoder–decoder approach, encodes a whole input sentence into a ﬁxed-length vector from which a translation will be decoded.
We conjectured that the use of a ﬁxed-length context vector is problematic for translating long sentences, based on a recent empirical study reported by Cho et al. (2014b) and Pouget-Abadie et al.
(2014).
In this paper, we proposed a novel architecture that addresses this issue. We extended the basic
encoder–decoder by letting a model (soft-)search for a set of input words, or their annotations computed by an encoder, when generating each target word. This frees the model from having to encode


a whole source sentence into a ﬁxed-length vector, and also lets the model focus only on information
relevant to the generation of the next target word. This has a major positive impact on the ability
of the neural machine translation system to yield good results on longer sentences. Unlike with
the traditional machine translation systems, all of the pieces of the translation system, including
the alignment mechanism, are jointly trained towards a better log-probability of producing correct
translations.
We tested the proposed model, called RNNsearch, on the task of English-to-French translation. The
experiment revealed that the proposed RNNsearch outperforms the conventional encoder–decoder
model (RNNencdec) signiﬁcantly, regardless of the sentence length and that it is much more robust to the length of a source sentence. From the qualitative analysis where we investigated the
(soft-)alignment generated by the RNNsearch, we were able to conclude that the model can correctly align each target word with the relevant words, or their annotations, in the source sentence as
it generated a correct translation.
Perhaps more importantly, the proposed approach achieved a translation performance comparable to
the existing phrase-based statistical machine translation. It is a striking result, considering that the
proposed architecture, or the whole family of neural machine translation, has only been proposed


the existing phrase-based statistical machine translation. It is a striking result, considering that the
proposed architecture, or the whole family of neural machine translation, has only been proposed
as recently as this year. We believe the architecture proposed here is a promising step toward better
machine translation and a better understanding of natural languages in general.
One of challenges left for the future is to better handle unknown, or rare words. This will be required
for the model to be more widely used and to match the performance of current state-of-the-art
machine translation systems in all contexts.
9
Published as a conference paper at ICLR 2015
ACKNOWLEDGMENTS
The authors would like to thank the developers of Theano (Bergstra et al. , 2010; Bastien et al. ,
2012). We acknowledge the support of the following agencies for research funding and computing
support: NSERC, Calcul Qu ´ebec, Compute Canada, the Canada Research Chairs and CIFAR. Bahdanau thanks the support from Planet Intelligent Systems GmbH. We also thank Felix Hill, Bart van
Merri ´enboer, Jean Pouget-Abadie, Coline Devin and Tae-Ho Kim.
REFERENCES
Axelrod, A., He, X., and Gao, J. (2011). Domain adaptation via pseudo in-domain data selection.


REFERENCES
Axelrod, A., He, X., and Gao, J. (2011). Domain adaptation via pseudo in-domain data selection.
InProceedings of the ACL Conference on Empirical Methods in Natural Language Processing
(EMNLP) , pages 355–362. Association for Computational Linguistics.
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N.,
and Bengio, Y . (2012). Theano: new features and speed improvements. Deep Learning and
Unsupervised Feature Learning NIPS 2012 Workshop.
Bengio, Y ., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient
descent is difﬁcult. IEEE Transactions on Neural Networks ,5(2), 157–166.
Bengio, Y ., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model.
J. Mach. Learn. Res. ,3, 1137–1155.


J. Mach. Learn. Res. ,3, 1137–1155.
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., WardeFarley, D., and Bengio, Y . (2010). Theano: a CPU and GPU math expression compiler. In
Proceedings of the Python for Scientiﬁc Computing Conference (SciPy) . Oral Presentation.
Boulanger-Lewandowski, N., Bengio, Y ., and Vincent, P. (2013). Audio chord recognition with
recurrent neural networks. In ISMIR .
Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y . (2014a).
Learning phrase representations using RNN encoder-decoder for statistical machine translation.
InProceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) . to
appear.
Cho, K., van Merri ¨enboer, B., Bahdanau, D., and Bengio, Y . (2014b). On the properties of neural
machine translation: Encoder–Decoder approaches. In Eighth Workshop on Syntax, Semantics


machine translation: Encoder–Decoder approaches. In Eighth Workshop on Syntax, Semantics
and Structure in Statistical Translation . to appear.
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast and robust
neural network joint models for statistical machine translation. In Association for Computational
Linguistics .
Forcada, M. L. and ˜Neco, R. P. (1997). Recursive hetero-associative memories for translation. In
J. Mira, R. Moreno-D ´ıaz, and J. Cabestany, editors, Biological and Artiﬁcial Computation: From
Neuroscience to Technology , volume 1240 of Lecture Notes in Computer Science , pages 453–462.
Springer Berlin Heidelberg.
Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y . (2013). Maxout networks. In Proceedings of The 30th International Conference on Machine Learning , pages 1319–
1327.
Graves, A. (2012). Sequence transduction with recurrent neural networks. In Proceedings of the
29th International Conference on Machine Learning (ICML 2012) .


1327.
Graves, A. (2012). Sequence transduction with recurrent neural networks. In Proceedings of the
29th International Conference on Machine Learning (ICML 2012) .
Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv: 1308.0850
[cs.NE] .
Graves, A., Jaitly, N., and Mohamed, A.-R. (2013). Hybrid speech recognition with deep bidirectional LSTM. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on , pages 273–278.
10
Published as a conference paper at ICLR 2015
Hermann, K. and Blunsom, P. (2014). Multilingual distributed representations without word alignment. In Proceedings of the Second International Conference on Learning Representations (ICLR
2014) .
Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut
f¨ur Informatik, Lehrstuhl Prof. Brauer, Technische Universit ¨at M ¨unchen.
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation ,9(8),
1735–1780.


Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation ,9(8),
1735–1780.
Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings
of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages
1700–1709. Association for Computational Linguistics.
Koehn, P. (2010). Statistical Machine Translation . Cambridge University Press, New York, NY ,
USA.
Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings
of the 2003 Conference of the North American Chapter of the Association for Computational
Linguistics on Human Language Technology - Volume 1 , NAACL ’03, pages 48–54, Stroudsburg,
PA, USA. Association for Computational Linguistics.
Pascanu, R., Mikolov, T., and Bengio, Y . (2013a). On the difﬁculty of training recurrent neural
networks. In ICML’2013 .
Pascanu, R., Mikolov, T., and Bengio, Y . (2013b). On the difﬁculty of training recurrent neural


Pascanu, R., Mikolov, T., and Bengio, Y . (2013b). On the difﬁculty of training recurrent neural
networks. In Proceedings of the 30th International Conference on Machine Learning (ICML
2013) .
Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y . (2014). How to construct deep recurrent neural
networks. In Proceedings of the Second International Conference on Learning Representations
(ICLR 2014) .
Pouget-Abadie, J., Bahdanau, D., van Merri ¨enboer, B., Cho, K., and Bengio, Y . (2014). Overcoming
the curse of sentence length for neural machine translation using automatic segmentation. In
Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation . to appear.
Schuster, M. and Paliwal, K. K. (1997). Bidirectional recurrent neural networks. Signal Processing,
IEEE Transactions on ,45(11), 2673–2681.
Schwenk, H. (2012). Continuous space translation models for phrase-based statistical machine
translation. In M. Kay and C. Boitet, editors, Proceedings of the 24th International Conference on


translation. In M. Kay and C. Boitet, editors, Proceedings of the 24th International Conference on
Computational Linguistics (COLIN) , pages 1071–1080. Indian Institute of Technology Bombay.
Schwenk, H., Dchelotte, D., and Gauvain, J.-L. (2006). Continuous space language models for
statistical machine translation. In Proceedings of the COLING/ACL on Main conference poster
sessions , pages 723–730. Association for Computational Linguistics.
Sutskever, I., Vinyals, O., and Le, Q. (2014). Sequence to sequence learning with neural networks.
InAdvances in Neural Information Processing Systems (NIPS 2014) .
Zeiler, M. D. (2012). ADADELTA: An adaptive learning rate method. arXiv: 1212.5701
[cs.LG] .
11
Published as a conference paper at ICLR 2015
A M ODEL ARCHITECTURE
A.1 A RCHITECTURAL CHOICES
The proposed scheme in Section 3 is a general framework where one can freely deﬁne, for instance,
the activation functions fof recurrent neural networks (RNN) and the alignment model a. Here, we
describe the choices we made for the experiments in this paper.


the activation functions fof recurrent neural networks (RNN) and the alignment model a. Here, we
describe the choices we made for the experiments in this paper.
A.1.1 R ECURRENT NEURAL NETWORK
For the activation function fof an RNN, we use the gated hidden unit recently proposed by Cho
et al. (2014a). The gated hidden unit is an alternative to the conventional simple units such as an
element-wise tanh . This gated unit is similar to a long short-term memory (LSTM) unit proposed
earlier by Hochreiter and Schmidhuber (1997), sharing with it the ability to better model and learn
long-term dependencies. This is made possible by having computation paths in the unfolded RNN
for which the product of derivatives is close to 1. These paths allow gradients to ﬂow backward
easily without suffering too much from the vanishing effect (Hochreiter, 1991; Bengio et al. , 1994;
Pascanu et al. , 2013a). It is therefore possible to use LSTM units instead of the gated hidden unit
described here, as was done in a similar context by Sutskever et al. (2014).
The new state siof the RNN employing ngated hidden units8is computed by


described here, as was done in a similar context by Sutskever et al. (2014).
The new state siof the RNN employing ngated hidden units8is computed by
si=f(si1;yi1;ci) = (1zi)si1+zi~si;
whereis an element-wise multiplication, and ziis the output of the update gates (see below). The
proposed updated state ~siis computed by
~si= tanh (We(yi1) +U[risi1] +Cci);
wheree(yi1)2Rmis anm-dimensional embedding of a word yi1, andriis the output of the
reset gates (see below). When yiis represented as a 1-of-Kvector,e(yi)is simply a column of an
embedding matrix E2RmK. Whenever possible, we omit bias terms to make the equations less
cluttered.
The update gates ziallow each hidden unit to maintain its previous activation, and the reset gates ri
control how much and what information from the previous state should be reset. We compute them
by
zi=(Wze(yi1) +Uzsi1+Czci);


by
zi=(Wze(yi1) +Uzsi1+Czci);
ri=(Wre(yi1) +Ursi1+Crci);
where()is a logistic sigmoid function.
At each step of the decoder, we compute the output probability (Eq. (4)) as a multi-layered function (Pascanu et al. , 2014). We use a single hidden layer of maxout units (Goodfellow et al. , 2013)
and normalize the output probabilities (one for each word) with a softmax function (see Eq. (6)).
A.1.2 A LIGNMENT MODEL
The alignment model should be designed considering that the model needs to be evaluated TxTy
times for each sentence pair of lengths TxandTy. In order to reduce computation, we use a singlelayer multilayer perceptron such that
a(si1;hj) =v>
atanh (Wasi1+Uahj);
whereWa2Rnn;Ua2Rn2nandva2Rnare the weight matrices. Since Uahjdoes not
depend oni, we can pre-compute it in advance to minimize the computational cost.


depend oni, we can pre-compute it in advance to minimize the computational cost.
8Here, we show the formula of the decoder. The same formula can be used in the encoder by simply
ignoring the context vector ciand the related terms.
12
Published as a conference paper at ICLR 2015
A.2 D ETAILED DESCRIPTION OF THE MODEL
A.2.1 E NCODER
In this section, we describe in detail the architecture of the proposed model (RNNsearch) used in the
experiments (see Sec. 4–5). From here on, we omit all bias terms in order to increase readability.
The model takes a source sentence of 1-of-K coded word vectors as input
x= (x1;:::;x Tx); xi2RKx
and outputs a translated sentence of 1-of-K coded word vectors
y= (y1;:::;y Ty); yi2RKy;
whereKxandKyare the vocabulary sizes of source and target languages, respectively. TxandTy
respectively denote the lengths of source and target sentences.
First, the forward states of the bidirectional recurrent neural network (BiRNN) are computed:
 !hi=(


respectively denote the lengths of source and target sentences.
First, the forward states of the bidirectional recurrent neural network (BiRNN) are computed:
 !hi=(
(1 !zi) !hi1+ !zi !hi, ifi>0
0 , ifi= 0
where
 !hi= tanh !WExi+ !Uh !ri !hi1i
 !zi= !WzExi+ !Uz !hi1
 !ri= !WrExi+ !Ur !hi1
:
E2RmKxis the word embedding matrix. !W; !Wz; !Wr2Rnm, !U; !Uz; !Ur2Rnnare
weight matrices. mandnare the word embedding dimensionality and the number of hidden units,
respectively. ()is as usual a logistic sigmoid function.
The backward states ( h1;; hTx)are computed similarly. We share the word embedding matrix
Ebetween the forward and backward RNNs, unlike the weight matrices.


Ebetween the forward and backward RNNs, unlike the weight matrices.
We concatenate the forward and backward states to to obtain the annotations (h1;h2;;hTx),
where
hi=" !hi hi#
(7)
A.2.2 D ECODER
The hidden state siof the decoder given the annotations from the encoder is computed by
si=(1zi)si1+zi~si;
where
~si= tanh (WEy i1+U[risi1] +Cci)
zi=(WzEyi1+Uzsi1+Czci)
ri=(WrEyi1+Ursi1+Crci)
Eis the word embedding matrix for the target language. W;W z;Wr2Rnm,U;Uz;Ur2Rnn,
andC;C z;Cr2Rn2nare weights. Again, mandnare the word embedding dimensionality
and the number of hidden units, respectively. The initial hidden state s0is computed by s0=
tanh
Ws h1
;whereWs2Rnn.


tanh
Ws h1
;whereWs2Rnn.
The context vector ciare recomputed at each step by the alignment model:
ci=TxX
j=1ijhj;
13
Published as a conference paper at ICLR 2015
Model Updates (105)Epochs Hours GPU Train NLL Dev. NLL
RNNenc-30 8.46 6.4 109 TITAN BLACK 28.1 53.0
RNNenc-50 6.00 4.5 108 Quadro K-6000 44.0 43.6
RNNsearch-30 4.71 3.6 113 TITAN BLACK 26.7 47.2
RNNsearch-50 2.88 2.2 111 Quadro K-6000 40.7 38.1
RNNsearch-50?6.67 5.0 252 Quadro K-6000 36.7 35.2
Table 2: Learning statistics and relevant information. Each update corresponds to updating the
parameters once using a single minibatch. One epoch is one pass through the training set. NLL is
the average conditional log-probabilities of the sentences in either the training set or the development
set. Note that the lengths of the sentences differ.
where
ij=exp (eij)PTx
k=1exp (eik)
eij=v>


set. Note that the lengths of the sentences differ.
where
ij=exp (eij)PTx
k=1exp (eik)
eij=v>
atanh (Wasi1+Uahj);
andhjis thej-th annotation in the source sentence (see Eq. (7)). va2Rn0;Wa2Rn0nand
Ua2Rn02nare weight matrices. Note that the model becomes RNN Encoder–Decoder (Cho
et al. , 2014a), if we ﬁx cito !hTx.
With the decoder state si1, the context ciand the last generated word yi1, we deﬁne the probability
of a target word yias
p(yijsi;yi1;ci)/exp
y>
iWoti
;
where
ti=
max~ti;2j1;~ti;2j	>
j=1;:::;l
and~ti;kis thek-th element of a vector ~tiwhich is computed by
~ti=Uosi1+VoEyi1+Coci:


and~ti;kis thek-th element of a vector ~tiwhich is computed by
~ti=Uosi1+VoEyi1+Coci:
Wo2RKyl,Uo2R2ln,Vo2R2lmandCo2R2l2nare weight matrices. This can be understood as having a deep output (Pascanu et al. , 2014) with a single maxout hidden layer (Goodfellow
et al. , 2013).
A.2.3 M ODEL SIZE
For all the models used in this paper, the size of a hidden layer nis 1000, the word embedding
dimensionality mis 620 and the size of the maxout hidden layer in the deep output lis 500. The
number of hidden units in the alignment model n0is 1000.
B T RAINING PROCEDURE
B.1 P ARAMETER INITIALIZATION
We initialized the recurrent weight matrices U;Uz;Ur; U; Uz; Ur; !U; !Uzand !Uras random orthogonal matrices. For WaandUa, we initialized them by sampling each element from the Gaussian


distribution of mean 0and variance 0:0012. All the elements of Vaand all the bias vectors were initialized to zero. Any other weight matrix was initialized by sampling from the Gaussian distribution
of mean 0and variance 0:012.
B.2 T RAINING
We used the stochastic gradient descent (SGD) algorithm. Adadelta (Zeiler, 2012) was used to
automatically adapt the learning rate of each parameter ( = 106and= 0:95). We explicitly
14
Published as a conference paper at ICLR 2015
normalized the L2-norm of the gradient of the cost function each time to be at most a predeﬁned
threshold of 1, when the norm was larger than the threshold (Pascanu et al. , 2013b). Each SGD
update direction was computed with a minibatch of 80 sentences.
At each update our implementation requires time proportional to the length of the longest sentence in
a minibatch. Hence, to minimize the waste of computation, before every 20-th update, we retrieved
1600 sentence pairs, sorted them according to the lengths and split them into 20 minibatches. The
training data was shufﬂed once before training and was traversed sequentially in this manner.
In Tables 2 we present the statistics related to training all the models used in the experiments.


In Tables 2 we present the statistics related to training all the models used in the experiments.
C T RANSLATIONS OF LONG SENTENCES
Source An admitting privilege is the right of a doctor to admit a patient to a hospital or a medical centre
to carry out a diagnosis or a procedure, based on his status as a health care worker at a hospital.
Reference Le privil `ege d’admission est le droit d’un m ´edecin, en vertu de son statut de membre soignant
d’un h ˆopital, d’admettre un patient dans un h ˆopital ou un centre m ´edical aﬁn d’y d ´elivrer un
diagnostic ou un traitement.
RNNenc-50 Un privil `ege d’admission est le droit d’un m ´edecin de reconna ˆıtre un patient `a l’h ˆopital ou un
centre m ´edical d’un diagnostic ou de prendre un diagnostic en fonction de son ´etat de sant ´e.


centre m ´edical d’un diagnostic ou de prendre un diagnostic en fonction de son ´etat de sant ´e.
RNNsearch-50 Un privil `ege d’admission est le droit d’un m ´edecin d’admettre un patient `a un h ˆopital ou un
centre m ´edical pour effectuer un diagnostic ou une proc ´edure, selon son statut de travailleur des
soins de sant ´e`a l’h ˆopital.
Google
TranslateUn privil `ege admettre est le droit d’un m ´edecin d’admettre un patient dans un h ˆopital ou un
centre m ´edical pour effectuer un diagnostic ou une proc ´edure, fond ´ee sur sa situation en tant
que travailleur de soins de sant ´e dans un h ˆopital.
Source This kind of experience is part of Disney’s efforts to ”extend the lifetime of its series and build
new relationships with audiences via digital platforms that are becoming ever more important,”
he added.


new relationships with audiences via digital platforms that are becoming ever more important,”
he added.
Reference Ce type d’exp ´erience entre dans le cadre des efforts de Disney pour ” ´etendre la dur ´ee de
vie de ses s ´eries et construire de nouvelles relations avec son public gr ˆace`a des plateformes
num´eriques qui sont de plus en plus importantes”, a-t-il ajout ´e.
RNNenc-50 Ce type d’exp ´erience fait partie des initiatives du Disney pour ”prolonger la dur ´ee de vie de
ses nouvelles et de d ´evelopper des liens avec les lecteurs num ´eriques qui deviennent plus complexes.
RNNsearch-50 Ce genre d’exp ´erience fait partie des efforts de Disney pour ”prolonger la dur ´ee de vie de ses
s´eries et cr ´eer de nouvelles relations avec des publics via des plateformes num ´eriques de plus
en plus importantes”, a-t-il ajout ´e.
Google


en plus importantes”, a-t-il ajout ´e.
Google
TranslateCe genre d’exp ´erience fait partie des efforts de Disney `a “´etendre la dur ´ee de vie de sa s ´erie et
construire de nouvelles relations avec le public par le biais des plates-formes num ´eriques qui
deviennent de plus en plus important”, at-il ajout ´e.
Source In a press conference on Thursday, Mr Blair stated that there was nothing in this video that might
constitute a ”reasonable motive” that could lead to criminal charges being brought against the
mayor.
Reference En conf ´erence de presse, jeudi, M. Blair a afﬁrm ´e qu’il n’y avait rien dans cette vid ´eo qui puisse
constituer des ”motifs raisonnables” pouvant mener au d ´epˆot d’une accusation criminelle contre
le maire.


le maire.
RNNenc-50 Lors de la conf ´erence de presse de jeudi, M. Blair a dit qu’il n’y avait rien dans cette vid ´eo qui
pourrait constituer une ”motivation raisonnable” pouvant entra ˆıner des accusations criminelles
port´ees contre le maire.
RNNsearch-50 Lors d’une conf ´erence de presse jeudi, M. Blair a d ´eclar ´e qu’il n’y avait rien dans cette vid ´eo qui
pourrait constituer un ”motif raisonnable” qui pourrait conduire `a des accusations criminelles
contre le maire.
Google
TranslateLors d’une conf ´erence de presse jeudi, M. Blair a d ´eclar ´e qu’il n’y avait rien dans cette vido
qui pourrait constituer un ”motif raisonnable” qui pourrait mener `a des accusations criminelles
portes contre le maire.


qui pourrait constituer un ”motif raisonnable” qui pourrait mener `a des accusations criminelles
portes contre le maire.
Table 3: The translations generated by RNNenc-50 and RNNsearch-50 from long source sentences
(30 words or more) selected from the test set. For each source sentence, we also show the goldstandard translation. The translations by Google Translate were made on 27 August 2014.
15


arXiv:1409.1556v6  [cs.CV]  10 Apr 2015Publishedasa conferencepaperat ICLR2015
VERYDEEPCONVOLUTIONAL NETWORKS
FORLARGE-SCALEIMAGERECOGNITION
KarenSimonyan∗& AndrewZisserman+
VisualGeometryGroup,DepartmentofEngineeringScience, UniversityofOxford
{karen,az }@robots.ox.ac.uk
ABSTRACT
In this work we investigate the effect of the convolutional n etwork depth on its
accuracy in the large-scale image recognition setting. Our main contribution is
a thorough evaluation of networks of increasing depth using an architecture with
verysmall( 3×3)convolutionﬁlters,whichshowsthatasigniﬁcantimprove ment
on the prior-art conﬁgurations can be achieved by pushing th e depth to 16–19
weight layers. These ﬁndings were the basis of our ImageNet C hallenge 2014
submission,whereourteamsecuredtheﬁrstandthesecondpl acesinthelocalisation and classiﬁcation tracks respectively. We also show th at our representations
generalise well to other datasets, where they achieve state -of-the-art results. We

Very Deep Convolutional Networks for Large-Scale Image Recognition

Summary:
In this work we investigate the effect of the convolutional network depth on
its accuracy in the large-scale image recognition setting. Our main
contribution is a thorough evaluation of networks of increasing depth using an
architecture with very small (3x3) convolution filters, which shows that a
significant improvement on the prior-art configurations can be achieved by
pushing the depth to 16-19 weight layers. These findings were the basis of our
ImageNet Challenge 2014 submission, where our team secured the first and the
second places in the localisation and classification tracks respectively. We
also show that our representations generalise well to other datasets, where
they achieve state-of-the-art results. We have made our two best-performing
ConvNet models publicly available to facilitate further research on the use of
deep visual representations in computer vision.

Material Recognition in the Wild with the Materials in Context Database

Bell, S., Upchurch, P., Snavely, N., Bala, K.

Return of the Devil in the Details: Delving Deep into Convolutional Nets

Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.

Deep Convolutional Filter Banks for Texture Recognition and Segmentation

Flexible, High Performance Convolutional Neural Networks for Image Classification

Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., Schmidhuber, J.


generalise well to other datasets, where they achieve state -of-the-art results. We
have made our two best-performingConvNet models publicly a vailable to facilitate furtherresearchontheuse ofdeepvisualrepresentati onsincomputervision.
1 INTRODUCTION
Convolutional networks (ConvNets) have recently enjoyed a great success in large-scale image and video recognition (Krizhevskyetal., 2012; Zeiler& Fergus, 2013; Sermanetet al., 2014;
Simonyan& Zisserman, 2014) which has become possible due to the large public image repositories,suchasImageNet(Denget al.,2009),andhigh-perform ancecomputingsystems,suchasGPUs
orlarge-scaledistributedclusters(Deanet al., 2012). In particular,animportantroleintheadvance
ofdeepvisualrecognitionarchitectureshasbeenplayedby theImageNetLarge-ScaleVisualRecognition Challenge (ILSVRC) (Russakovskyet al., 2014), whic h has served as a testbed for a few


generationsof large-scale image classiﬁcation systems, f rom high-dimensionalshallow feature encodings(Perronninetal.,2010)(thewinnerofILSVRC-2011 )todeepConvNets(Krizhevskyet al.,
2012)(thewinnerofILSVRC-2012).
With ConvNets becoming more of a commodity in the computer vi sion ﬁeld, a number of attempts have been made to improve the original architecture o f Krizhevskyet al. (2012) in a
bid to achieve better accuracy. For instance, the best-perf orming submissions to the ILSVRC2013 (Zeiler&Fergus, 2013; Sermanetetal., 2014) utilised smaller receptive window size and
smaller stride of the ﬁrst convolutional layer. Another lin e of improvements dealt with training
and testing the networks densely over the whole image and ove r multiple scales (Sermanetet al.,
2014; Howard, 2014). In this paper, we address another impor tant aspect of ConvNet architecture
design–itsdepth. Tothisend,we ﬁxotherparametersofthea rchitecture,andsteadilyincreasethe
depth of the network by adding more convolutionallayers, wh ich is feasible due to the use of very


depth of the network by adding more convolutionallayers, wh ich is feasible due to the use of very
small (3×3)convolutionﬁltersinall layers.
As a result, we come up with signiﬁcantly more accurate ConvN et architectures, which not only
achieve the state-of-the-art accuracy on ILSVRC classiﬁca tion and localisation tasks, but are also
applicabletootherimagerecognitiondatasets,wherethey achieveexcellentperformanceevenwhen
usedasa partofa relativelysimple pipelines(e.g.deepfea turesclassiﬁed byalinearSVM without
ﬁne-tuning). We havereleasedourtwobest-performingmode ls1tofacilitatefurtherresearch.
The rest of the paper is organised as follows. In Sect. 2, we de scribe our ConvNet conﬁgurations.
The details of the image classiﬁcation trainingand evaluat ionare then presented in Sect. 3, and the
∗current afﬁliation: Google DeepMind+current afﬁliation: Universityof Oxfordand Google DeepMi nd
1http://www.robots.ox.ac.uk/ ˜vgg/research/very_deep/
1


1http://www.robots.ox.ac.uk/ ˜vgg/research/very_deep/
1
Publishedasa conferencepaperat ICLR2015
conﬁgurations are compared on the ILSVRC classiﬁcation tas k in Sect. 4. Sect. 5 concludes the
paper. For completeness,we also describeand assess our ILS VRC-2014object localisationsystem
inAppendixA,anddiscussthegeneralisationofverydeepfe aturestootherdatasetsinAppendixB.
Finally,AppendixCcontainsthelist ofmajorpaperrevisio ns.
2 CONVNETCONFIGURATIONS
To measure the improvement brought by the increased ConvNet depth in a fair setting, all our
ConvNet layer conﬁgurations are designed using the same pri nciples, inspired by Ciresan etal.
(2011); Krizhevskyet al. (2012). In this section, we ﬁrst de scribe a generic layout of our ConvNet
conﬁgurations(Sect.2.1)andthendetailthespeciﬁcconﬁg urationsusedintheevaluation(Sect.2.2).


Ourdesignchoicesarethendiscussedandcomparedtothepri orart inSect. 2.3.
2.1 A RCHITECTURE
During training, the input to our ConvNets is a ﬁxed-size 224×224RGB image. The only preprocessingwedoissubtractingthemeanRGBvalue,computed onthetrainingset,fromeachpixel.
Theimageispassedthroughastackofconvolutional(conv.) layers,whereweuseﬁlterswithavery
small receptive ﬁeld: 3×3(which is the smallest size to capture the notion of left/rig ht, up/down,
center). In one of the conﬁgurationswe also utilise 1×1convolutionﬁlters, which can be seen as
a linear transformationof the input channels (followed by n on-linearity). The convolutionstride is
ﬁxedto1pixel;thespatialpaddingofconv.layerinputissuchthatt hespatialresolutionispreserved
afterconvolution,i.e. the paddingis 1pixel for3×3conv.layers. Spatial poolingis carriedoutby


afterconvolution,i.e. the paddingis 1pixel for3×3conv.layers. Spatial poolingis carriedoutby
ﬁvemax-poolinglayers,whichfollowsomeoftheconv.layer s(notalltheconv.layersarefollowed
bymax-pooling). Max-poolingisperformedovera 2×2pixelwindow,withstride 2.
Astackofconvolutionallayers(whichhasadifferentdepth indifferentarchitectures)isfollowedby
three Fully-Connected(FC) layers: the ﬁrst two have4096ch annelseach,the thirdperforms1000way ILSVRC classiﬁcation and thus contains1000channels(o ne foreach class). The ﬁnal layer is
thesoft-maxlayer. Theconﬁgurationofthefullyconnected layersis thesameinall networks.
Allhiddenlayersareequippedwiththerectiﬁcation(ReLU( Krizhevskyetal.,2012))non-linearity.
We note that none of our networks (except for one) contain Loc al Response Normalisation
(LRN) normalisation (Krizhevskyet al., 2012): as will be sh own in Sect. 4, such normalisation


(LRN) normalisation (Krizhevskyet al., 2012): as will be sh own in Sect. 4, such normalisation
does not improve the performance on the ILSVRC dataset, but l eads to increased memory consumption and computation time. Where applicable, the param eters for the LRN layer are those
of(Krizhevskyetal., 2012).
2.2 C ONFIGURATIONS
The ConvNet conﬁgurations, evaluated in this paper, are out lined in Table 1, one per column. In
the following we will refer to the nets by their names (A–E). A ll conﬁgurationsfollow the generic
design presented in Sect. 2.1, and differ only in the depth: f rom 11 weight layers in the network A
(8conv.and3FClayers)to19weightlayersinthenetworkE(1 6conv.and3FClayers). Thewidth
of conv.layers (the number of channels) is rather small, sta rting from 64in the ﬁrst layer and then
increasingbyafactorof 2aftereachmax-poolinglayer,untilit reaches 512.
In Table 2 we reportthe numberof parametersfor each conﬁgur ation. In spite of a large depth, the


In Table 2 we reportthe numberof parametersfor each conﬁgur ation. In spite of a large depth, the
numberof weights in our netsis not greater thanthe numberof weightsin a moreshallow net with
largerconv.layerwidthsandreceptiveﬁelds(144Mweights in(Sermanetet al., 2014)).
2.3 D ISCUSSION
Our ConvNet conﬁgurations are quite different from the ones used in the top-performing entries
of the ILSVRC-2012 (Krizhevskyetal., 2012) and ILSVRC-201 3 competitions (Zeiler& Fergus,
2013;Sermanetet al.,2014). Ratherthanusingrelativelyl argereceptiveﬁeldsintheﬁrstconv.layers(e.g.11×11withstride 4in(Krizhevskyet al.,2012),or 7×7withstride 2in(Zeiler& Fergus,
2013; Sermanetet al., 2014)), we use very small 3×3receptive ﬁelds throughout the whole net,
whichareconvolvedwiththeinputateverypixel(withstrid e1). Itiseasytoseethatastackoftwo


whichareconvolvedwiththeinputateverypixel(withstrid e1). Itiseasytoseethatastackoftwo
3×3conv.layers(withoutspatialpoolinginbetween)hasaneff ectivereceptiveﬁeldof 5×5;three
2
Publishedasa conferencepaperat ICLR2015
Table 1:ConvNet conﬁgurations (shown in columns). The depth of the conﬁgurations increase s
fromtheleft(A)totheright(E),asmorelayersareadded(th eaddedlayersareshowninbold). The
convolutional layer parameters are denoted as “conv /an}bracketle{treceptive ﬁeld size /an}bracketri}ht-/an}bracketle{tnumber of channels /an}bracketri}ht”.
TheReLU activationfunctionisnotshownforbrevity.
ConvNet Conﬁguration
A A-LRN B C D E
11weight 11weight 13 weight 16weight 16weight 19 weight
layers layers layers layers layers layers
input (224×224RGBimage)
conv3-64 conv3-64 conv3-64 conv3-64 conv3-64 conv3-64


conv3-512 conv3-512 conv3-512 conv3-512 conv3-512 conv3-512
conv1-512 conv3-512 conv3-512
conv3-512
maxpool
FC-4096
FC-4096
FC-1000
soft-max
Table2:Number ofparameters (inmillions).
Network A,A-LRN BCDE
Number of parameters 133 133134138144
such layers have a 7×7effectivereceptive ﬁeld. So what have we gainedby using, fo r instance, a
stackofthree 3×3conv.layersinsteadofasingle 7×7layer? First,weincorporatethreenon-linear
rectiﬁcation layers instead of a single one, which makes the decision functionmore discriminative.
Second, we decrease the number of parameters: assuming that both the input and the output of a
three-layer 3×3convolutionstack has Cchannels,the stack is parametrisedby 3/parenleftbig
32C2/parenrightbig
= 27C2
weights; at the same time, a single 7×7conv. layer would require 72C2= 49C2parameters, i.e.


= 27C2
weights; at the same time, a single 7×7conv. layer would require 72C2= 49C2parameters, i.e.
81%more. Thiscan be seen as imposinga regularisationon the 7×7conv.ﬁlters, forcingthemto
haveadecompositionthroughthe 3×3ﬁlters(withnon-linearityinjectedin between).
The incorporation of 1×1conv. layers (conﬁguration C, Table 1) is a way to increase th e nonlinearity of the decision function without affecting the re ceptive ﬁelds of the conv. layers. Even
thoughinourcasethe 1×1convolutionisessentiallyalinearprojectionontothespa ceofthesame
dimensionality(thenumberofinputandoutputchannelsist hesame),anadditionalnon-linearityis
introducedbytherectiﬁcationfunction. Itshouldbenoted that1×1conv.layershaverecentlybeen
utilisedin the“NetworkinNetwork”architectureofLinet a l.(2014).
Small-size convolution ﬁlters have been previously used by Ciresan etal. (2011), but their nets


Small-size convolution ﬁlters have been previously used by Ciresan etal. (2011), but their nets
are signiﬁcantly less deep than ours, and they did not evalua te on the large-scale ILSVRC
dataset. Goodfellowet al. (2014) applied deep ConvNets ( 11weight layers) to the task of
street number recognition, and showed that the increased de pth led to better performance.
GoogLeNet(Szegedyet al., 2014), a top-performingentryof the ILSVRC-2014classiﬁcation task,
was developed independentlyof our work, but is similar in th at it is based on very deep ConvNets
3
Publishedasa conferencepaperat ICLR2015
(22 weight layers) and small convolution ﬁlters (apart from 3×3, they also use 1×1and5×5
convolutions). Their network topology is, however, more co mplex than ours, and the spatial resolution of the feature maps is reduced more aggressively in th e ﬁrst layers to decrease the amount
of computation. As will be shown in Sect. 4.5, our model is out performing that of Szegedyetal.


of computation. As will be shown in Sect. 4.5, our model is out performing that of Szegedyetal.
(2014)intermsofthesingle-networkclassiﬁcationaccura cy.
3 CLASSIFICATION FRAMEWORK
In the previous section we presented the details of our netwo rk conﬁgurations. In this section, we
describethe detailsofclassiﬁcationConvNettrainingand evaluation.
3.1 T RAINING
The ConvNet training procedure generally follows Krizhevs kyetal. (2012) (except for sampling
theinputcropsfrommulti-scaletrainingimages,asexplai nedlater). Namely,thetrainingiscarried
out by optimising the multinomial logistic regression obje ctive using mini-batch gradient descent
(based on back-propagation(LeCunet al., 1989)) with momen tum. The batch size was set to 256,
momentum to 0.9. The training was regularised by weight decay (the L2penalty multiplier set to
5·10−4)anddropoutregularisationfortheﬁrsttwofully-connect edlayers(dropoutratiosetto 0.5).


5·10−4)anddropoutregularisationfortheﬁrsttwofully-connect edlayers(dropoutratiosetto 0.5).
Thelearningrate wasinitially setto 10−2,andthendecreasedbyafactorof 10whenthevalidation
set accuracy stopped improving. In total, the learning rate was decreased 3 times, and the learning
was stopped after 370K iterations (74 epochs). We conjecture that in spite of the l arger number of
parametersandthegreaterdepthofournetscomparedto(Kri zhevskyetal.,2012),thenetsrequired
lessepochstoconvergedueto(a)implicitregularisationi mposedbygreaterdepthandsmallerconv.
ﬁlter sizes; (b)pre-initialisationofcertainlayers.
The initialisation of the networkweightsis important,sin ce bad initialisation can stall learningdue
to the instability of gradient in deep nets. To circumvent th is problem, we began with training
the conﬁgurationA (Table 1), shallow enoughto be trained wi th randominitialisation. Then,when


the conﬁgurationA (Table 1), shallow enoughto be trained wi th randominitialisation. Then,when
trainingdeeperarchitectures,weinitialisedtheﬁrstfou rconvolutionallayersandthelastthreefullyconnectedlayerswiththelayersofnetA(theintermediatel ayerswereinitialisedrandomly). Wedid
notdecreasethelearningrateforthepre-initialisedlaye rs,allowingthemtochangeduringlearning.
For random initialisation (where applicable), we sampled t he weights from a normal distribution
with thezeromeanand 10−2variance. The biaseswere initialisedwith zero. It isworth notingthat
after the paper submission we found that it is possible to ini tialise the weights without pre-training
byusingthe randominitialisationprocedureofGlorot&Ben gio(2010).
Toobtaintheﬁxed-size 224×224ConvNetinputimages,theywererandomlycroppedfromresca led
training images (one crop per image per SGD iteration). To fu rther augment the training set, the
cropsunderwentrandomhorizontalﬂippingandrandomRGBco lourshift(Krizhevskyet al.,2012).
Trainingimagerescalingisexplainedbelow.


Trainingimagerescalingisexplainedbelow.
Training image size. LetSbe the smallest side of an isotropically-rescaledtraining image, from
which the ConvNet input is cropped (we also refer to Sas the training scale). While the crop size
is ﬁxed to 224×224, in principle Scan take on any value not less than 224: forS= 224the crop
will capture whole-image statistics, completely spanning the smallest side of a training image; for
S≫224thecropwillcorrespondtoasmallpartoftheimage,contain ingasmallobjectoranobject
part.
We considertwoapproachesforsettingthetrainingscale S. Theﬁrst istoﬁx S,whichcorresponds
to single-scale training (note that image content within th e sampled crops can still represent multiscale image statistics). In our experiments, we evaluated m odels trained at two ﬁxed scales: S=
256(which has been widely used in the prior art (Krizhevskyet al ., 2012; Zeiler&Fergus, 2013;
Sermanetet al., 2014)) and S= 384. Given a ConvNet conﬁguration,we ﬁrst trained the network


Sermanetet al., 2014)) and S= 384. Given a ConvNet conﬁguration,we ﬁrst trained the network
usingS= 256. To speed-up training of the S= 384network, it was initialised with the weights
pre-trainedwith S= 256,andwe useda smallerinitiallearningrateof 10−3.
The second approachto setting Sis multi-scale training, where each training image is indiv idually
rescaled by randomly sampling Sfrom a certain range [Smin,Smax](we used Smin= 256and
Smax= 512). Sinceobjectsinimagescanbeofdifferentsize,itisbene ﬁcialtotakethisintoaccount
duringtraining. Thiscanalso beseen astrainingset augmen tationbyscale jittering,wherea single
4
Publishedasa conferencepaperat ICLR2015
model is trained to recognise objects over a wide range of sca les. For speed reasons, we trained
multi-scale models by ﬁne-tuning all layers of a single-sca le model with the same conﬁguration,
pre-trainedwithﬁxed S= 384.
3.2 T ESTING


pre-trainedwithﬁxed S= 384.
3.2 T ESTING
Attest time,givena trainedConvNetandaninputimage,itis classiﬁed inthefollowingway. First,
it is isotropically rescaled to a pre-deﬁned smallest image side, denoted as Q(we also refer to it
as the test scale). We note that Qis not necessarily equal to the training scale S(as we will show
in Sect. 4, usingseveralvaluesof QforeachSleadsto improvedperformance). Then,the network
is applied densely overthe rescaled test image in a way simil ar to (Sermanetet al., 2014). Namely,
the fully-connected layers are ﬁrst converted to convoluti onal layers (the ﬁrst FC layer to a 7×7
conv. layer, the last two FC layers to 1×1conv. layers). The resulting fully-convolutional net is
then applied to the whole (uncropped) image. The result is a c lass score map with the number of
channels equal to the number of classes, and a variable spati al resolution, dependent on the input
imagesize. Finally,toobtainaﬁxed-sizevectorofclasssc oresfortheimage,theclassscoremapis


imagesize. Finally,toobtainaﬁxed-sizevectorofclasssc oresfortheimage,theclassscoremapis
spatially averaged(sum-pooled). We also augmentthe test s et by horizontalﬂippingof the images;
thesoft-maxclassposteriorsoftheoriginalandﬂippedima gesareaveragedtoobtaintheﬁnalscores
fortheimage.
Since the fully-convolutionalnetwork is applied over the w hole image, there is no need to sample
multiple crops at test time (Krizhevskyetal., 2012), which is less efﬁcient as it requires network
re-computationforeachcrop. Atthesametime,usingalarge setofcrops,asdonebySzegedyetal.
(2014),canleadtoimprovedaccuracy,asit resultsin aﬁner samplingoftheinputimagecompared
tothefully-convolutionalnet. Also,multi-cropevaluati oniscomplementarytodenseevaluationdue
to different convolution boundary conditions: when applyi ng a ConvNet to a crop, the convolved
feature mapsare paddedwith zeros, while in the case of dense evaluationthe paddingfor the same


feature mapsare paddedwith zeros, while in the case of dense evaluationthe paddingfor the same
crop naturally comes from the neighbouring parts of an image (due to both the convolutions and
spatial pooling), which substantially increases the overa ll network receptive ﬁeld, so more context
iscaptured. Whilewebelievethatinpracticetheincreased computationtimeofmultiplecropsdoes
notjustifythepotentialgainsinaccuracy,forreferencew ealsoevaluateournetworksusing 50crops
perscale( 5×5regulargridwith 2ﬂips),foratotalof 150cropsover 3scales,whichiscomparable
to144cropsover 4scalesusedbySzegedyetal. (2014).
3.3 IMPLEMENTATION DETAILS
OurimplementationisderivedfromthepubliclyavailableC ++ Caffetoolbox(Jia,2013)(branched
out in December 2013), but contains a number of signiﬁcant mo diﬁcations, allowing us to perform
trainingandevaluationonmultipleGPUsinstalledinasing lesystem,aswellastrainandevaluateon
full-size (uncropped) images at multiple scales (as descri bed above). Multi-GPU training exploits


full-size (uncropped) images at multiple scales (as descri bed above). Multi-GPU training exploits
data parallelism, and is carried out by splitting each batch of training images into several GPU
batches, processed in parallel on each GPU. After the GPU bat ch gradientsare computed, they are
averaged to obtain the gradient of the full batch. Gradient c omputation is synchronous across the
GPUs, sothe resultisexactlythesame aswhentrainingona si ngleGPU.
While more sophisticated methods of speeding up ConvNet tra ining have been recently proposed (Krizhevsky, 2014), which employmodeland data paral lelism for differentlayersof the net,
wehavefoundthatourconceptuallymuchsimplerschemealre adyprovidesaspeedupof 3.75times
on an off-the-shelf4-GPU system, as comparedto using a sing le GPU. On a system equippedwith
fourNVIDIATitanBlackGPUs,trainingasinglenettook2–3w eeksdependingonthearchitecture.
4 CLASSIFICATION EXPERIMENTS
Dataset. In this section, we present the image classiﬁcation results achieved by the described


4 CLASSIFICATION EXPERIMENTS
Dataset. In this section, we present the image classiﬁcation results achieved by the described
ConvNetarchitecturesontheILSVRC-2012dataset(whichwa susedforILSVRC2012–2014challenges). The dataset includes images of 1000 classes, and is split into three sets: training ( 1.3M
images), validation ( 50K images), and testing ( 100K images with held-out class labels). The classiﬁcation performanceis evaluated using two measures: the top-1 and top-5 error. The former is a
multi-class classiﬁcation error, i.e. the proportion of in correctly classiﬁed images; the latter is the
5
Publishedasa conferencepaperat ICLR2015
main evaluation criterion used in ILSVRC, and is computed as the proportion of images such that
theground-truthcategoryisoutsidethetop-5predictedca tegories.
Forthemajorityofexperiments,weusedthevalidationseta sthetestset. Certainexperimentswere
also carried out on the test set and submitted to the ofﬁcial I LSVRC server as a “VGG” team entry
tothe ILSVRC-2014competition(Russakovskyet al., 2014).


tothe ILSVRC-2014competition(Russakovskyet al., 2014).
4.1 SINGLESCALEEVALUATION
We begin with evaluating the performanceof individual Conv Net models at a single scale with the
layerconﬁgurationsdescribedin Sect. 2.2. The test images ize was set as follows: Q=Sforﬁxed
S,andQ= 0.5(Smin+Smax)forjittered S∈[Smin,Smax]. Theresultsofareshownin Table3.
First, we note that using local response normalisation (A-L RN network) does not improve on the
model A without any normalisation layers. We thus do not empl oy normalisation in the deeper
architectures(B–E).
Second, we observe that the classiﬁcation error decreases w ith the increased ConvNet depth: from
11 layers in A to 19 layers in E. Notably, in spite of the same de pth, the conﬁguration C (which
containsthree 1×1conv.layers),performsworsethantheconﬁgurationD,whic huses3×3conv.
layersthroughoutthenetwork. Thisindicatesthatwhileth e additionalnon-linearitydoeshelp(Cis


layersthroughoutthenetwork. Thisindicatesthatwhileth e additionalnon-linearitydoeshelp(Cis
better than B), it is also important to capture spatial conte xt by using conv. ﬁlters with non-trivial
receptive ﬁelds (D is better than C). The error rate of our arc hitecture saturates when the depth
reaches19layers,butevendeepermodelsmightbebeneﬁcialforlarger datasets. Wealsocompared
the net B with a shallow net with ﬁve 5×5conv. layers, which was derived from B by replacing
eachpairof 3×3conv. layerswithasingle 5×5conv. layer(whichhasthesamereceptiveﬁeldas
explained in Sect. 2.3). The top-1 error of the shallow net wa s measured to be 7%higher than that
of B (on a center crop),which conﬁrmsthat a deepnet with smal l ﬁlters outperformsa shallow net
withlargerﬁlters.
Finally, scale jittering at training time ( S∈[256;512] ) leads to signiﬁcantly better results than


Finally, scale jittering at training time ( S∈[256;512] ) leads to signiﬁcantly better results than
training on images with ﬁxed smallest side ( S= 256orS= 384), even though a single scale is
usedattesttime. Thisconﬁrmsthattrainingsetaugmentati onbyscalejitteringisindeedhelpfulfor
capturingmulti-scaleimagestatistics.
Table3:ConvNetperformanceatasingle testscale.
ConvNet conﬁg. (Table 1) smallest image side top-1 val.error (%) top-5 val.error (%)
train(S)test (Q)
A 256 256 29.6 10.4
A-LRN 256 256 29.7 10.5
B 256 256 28.7 9.9
C256 256 28.1 9.4
384 384 28.1 9.3
[256;512] 384 27.3 8.8
D256 256 27.0 8.8
384 384 26.8 8.7
[256;512] 384 25.6 8.1
E256 256 27.3 9.0
384 384 26.9 8.7
[256;512] 384 25.5 8.0
4.2 M ULTI-SCALEEVALUATION


384 384 26.9 8.7
[256;512] 384 25.5 8.0
4.2 M ULTI-SCALEEVALUATION
HavingevaluatedtheConvNetmodelsatasinglescale,wenow assesstheeffectofscalejitteringat
testtime. Itconsistsofrunningamodeloverseveralrescal edversionsofatestimage(corresponding
to different values of Q), followed by averaging the resulting class posteriors. Co nsidering that a
large discrepancy between training and testing scales lead s to a drop in performance, the models
trained with ﬁxed Swere evaluated over three test image sizes, close to the trai ning one: Q=
{S−32,S,S+ 32}. At the same time, scale jittering at training time allows th e network to be
appliedto a widerrangeofscales at test time,so the modeltr ainedwithvariable S∈[Smin;Smax]
wasevaluatedoveralargerrangeofsizes Q={Smin,0.5(Smin+Smax),Smax}.
6
Publishedasa conferencepaperat ICLR2015
Theresults,presentedinTable4,indicatethatscalejitte ringattest timeleadstobetterperformance


6
Publishedasa conferencepaperat ICLR2015
Theresults,presentedinTable4,indicatethatscalejitte ringattest timeleadstobetterperformance
(as compared to evaluating the same model at a single scale, s hown in Table 3). As before, the
deepest conﬁgurations(D and E) perform the best, and scale j ittering is better than training with a
ﬁxed smallest side S. Our best single-network performance on the validation set is24.8%/7.5%
top-1/top-5error(highlightedinboldinTable4). Onthete stset,theconﬁgurationEachieves 7.3%
top-5error.
Table4:ConvNetperformanceatmultiple test scales.
ConvNet conﬁg. (Table 1) smallest image side top-1val. error (%) top-5val. error (%)
train(S)test(Q)
B 256 224,256,288 28.2 9.6
C256 224,256,288 27.7 9.2
384 352,384,416 27.8 9.2
[256;512] 256,384,512 26.3 8.2
D256 224,256,288 26.6 8.6


[256;512] 256,384,512 26.3 8.2
D256 224,256,288 26.6 8.6
384 352,384,416 26.5 8.6
[256;512] 256,384,512 24.8 7.5
E256 224,256,288 26.9 8.7
384 352,384,416 26.7 8.6
[256;512] 256,384,512 24.8 7.5
4.3 M ULTI-CROP EVALUATION
In Table 5 we compare dense ConvNet evaluation with mult-cro p evaluation (see Sect. 3.2 for details). We also assess the complementarityof thetwo evalua tiontechniquesbyaveragingtheirsoftmax outputs. As can be seen, using multiple crops performs sl ightly better than dense evaluation,
andthe two approachesareindeedcomplementary,astheir co mbinationoutperformseach ofthem.
As noted above, we hypothesize that this is due to a different treatment of convolution boundary
conditions.
Table 5:ConvNetevaluationtechniques comparison. Inall experimentsthe trainingscale Swas
sampledfrom [256;512] ,andthreetest scales Qwereconsidered: {256,384,512}.


sampledfrom [256;512] ,andthreetest scales Qwereconsidered: {256,384,512}.
ConvNet conﬁg. (Table 1) Evaluationmethod top-1 val. error(%) top-5 val. error (%)
Ddense 24.8 7.5
multi-crop 24.6 7.5
multi-crop &dense 24.4 7.2
Edense 24.8 7.5
multi-crop 24.6 7.4
multi-crop &dense 24.4 7.1
4.4 C ONVNETFUSION
Upuntilnow,weevaluatedtheperformanceofindividualCon vNetmodels. Inthispartoftheexperiments,wecombinetheoutputsofseveralmodelsbyaveragin gtheirsoft-maxclassposteriors. This
improvesthe performancedueto complementarityof the mode ls, andwas used in the top ILSVRC
submissions in 2012 (Krizhevskyet al., 2012) and 2013 (Zeil er&Fergus, 2013; Sermanetet al.,
2014).
The results are shown in Table 6. By the time of ILSVRC submiss ion we had only trained the
single-scale networks, as well as a multi-scale model D (by ﬁ ne-tuning only the fully-connected


single-scale networks, as well as a multi-scale model D (by ﬁ ne-tuning only the fully-connected
layers rather than all layers). The resulting ensemble of 7 n etworks has 7.3%ILSVRC test error.
After the submission, we considered an ensemble of only two b est-performing multi-scale models
(conﬁgurations D and E), which reduced the test error to 7.0%using dense evaluation and 6.8%
using combined dense and multi-crop evaluation. For refere nce, our best-performingsingle model
achieves7.1%error(modelE, Table5).
4.5 C OMPARISON WITH THE STATE OF THE ART
Finally, we compare our results with the state of the art in Ta ble 7. In the classiﬁcation task of
ILSVRC-2014 challenge (Russakovskyet al., 2014), our “VGG ” team secured the 2nd place with
7
Publishedasa conferencepaperat ICLR2015
Table6:Multiple ConvNetfusion results.
Combined ConvNet modelsError
top-1 val top-5val top-5test
ILSVRCsubmission


7.3%test errorusinganensembleof7 models. Afterthesubmissio n,we decreasedtheerrorrateto
6.8%usinganensembleof2models.
As can be seen from Table 7, our very deep ConvNetssigniﬁcant ly outperformthe previousgeneration of models, which achieved the best results in the ILSVR C-2012 and ILSVRC-2013 competitions. Our result is also competitivewith respect to the cla ssiﬁcation task winner(GoogLeNetwith
6.7%error) and substantially outperforms the ILSVRC-2013 winn ing submission Clarifai, which
achieved 11.2%with outside training data and 11.7%without it. This is remarkable, considering
that our best result is achievedby combiningjust two models – signiﬁcantly less than used in most
ILSVRC submissions. In terms of the single-net performance , our architecture achieves the best
result (7.0%test error), outperforming a single GoogLeNet by 0.9%. Notably, we did not depart
from the classical ConvNet architecture of LeCunetal. (198 9), but improved it by substantially
increasingthedepth.


from the classical ConvNet architecture of LeCunetal. (198 9), but improved it by substantially
increasingthedepth.
Table 7:Comparison with the state of the art in ILSVRC classiﬁcation . Our methodis denoted
as“VGG”.Onlytheresultsobtainedwithoutoutsidetrainin gdataarereported.
Method top-1 val. error(%) top-5val. error (%) top-5testerror (%)
VGG(2nets, multi-crop& dense eval.) 23.7 6.8 6.8
VGG(1net, multi-crop& dense eval.) 24.4 7.1 7.0
VGG(ILSVRCsubmission, 7nets, dense eval.) 24.7 7.5 7.3
GoogLeNet (Szegedy et al., 2014) (1net) - 7.9
GoogLeNet (Szegedy et al., 2014) (7nets) - 6.7
MSRA(He et al., 2014) (11nets) - - 8.1
MSRA(He et al., 2014) (1net) 27.9 9.1 9.1
Clarifai(Russakovsky et al., 2014) (multiplenets) - - 11.7


Clarifai(Russakovsky et al., 2014) (multiplenets) - - 11.7
Clarifai(Russakovsky et al., 2014) (1net) - - 12.5
Zeiler& Fergus (Zeiler&Fergus, 2013) (6nets) 36.0 14.7 14.8
Zeiler& Fergus (Zeiler&Fergus, 2013) (1net) 37.5 16.0 16.1
OverFeat (Sermanetet al.,2014) (7nets) 34.0 13.2 13.6
OverFeat (Sermanetet al.,2014) (1net) 35.7 14.2 Krizhevsky et al.(Krizhevsky et al., 2012) (5nets) 38.1 16.4 16.4
Krizhevsky et al.(Krizhevsky et al., 2012) (1net) 40.7 18.2 5 CONCLUSION
In this work we evaluated very deep convolutional networks ( up to 19 weight layers) for largescale image classiﬁcation. It was demonstrated that the rep resentation depth is beneﬁcial for the
classiﬁcationaccuracy,andthatstate-of-the-artperfor manceontheImageNetchallengedatasetcan


classiﬁcationaccuracy,andthatstate-of-the-artperfor manceontheImageNetchallengedatasetcan
beachievedusingaconventionalConvNetarchitecture(LeC unet al.,1989;Krizhevskyet al.,2012)
withsubstantiallyincreaseddepth. Intheappendix,weals oshowthatourmodelsgeneralisewellto
a wide range of tasks and datasets, matchingor outperformin gmore complexrecognitionpipelines
builtaroundlessdeepimagerepresentations. Ourresultsy etagainconﬁrmtheimportanceof depth
invisualrepresentations.
ACKNOWLEDGEMENTS
ThisworkwassupportedbyERCgrantVisRecno.228180. Wegra tefullyacknowledgethesupport
ofNVIDIACorporationwiththedonationoftheGPUsusedfort hisresearch.
8
Publishedasa conferencepaperat ICLR2015
REFERENCES
Bell, S., Upchurch, P.,Snavely, N., and Bala, K. Material re cognition inthe wild withthe materials in context
database. CoRR,abs/1412.0623, 2014.


database. CoRR,abs/1412.0623, 2014.
Chatﬁeld, K., Simonyan, K., Vedaldi, A., and Zisserman, A. R eturn of the devil in the details: Delving deep
intoconvolutional nets. In Proc.BMVC. ,2014.
Cimpoi,M.,Maji,S.,andVedaldi,A. Deepconvolutionalﬁlt erbanksfortexturerecognitionandsegmentation.
CoRR,abs/1411.6836, 2014.
Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., and Schmidhuber, J. Flexible, high performance
convolutional neural networks for image classiﬁcation. In IJCAI,pp. 1237–1242, 2011.
Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M. , Ranzato, M., Senior, A., Tucker, P., Yang,
K.,Le,Q. V.,andNg, A.Y. Large scale distributeddeepnetwo rks. InNIPS,pp. 1232–1240, 2012.


K.,Le,Q. V.,andNg, A.Y. Large scale distributeddeepnetwo rks. InNIPS,pp. 1232–1240, 2012.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei , L. Imagenet: A large-scale hierarchical image
database. In Proc.CVPR ,2009.
Donahue,J.,Jia,Y.,Vinyals,O.,Hoffman,J.,Zhang,N.,Tz eng,E.,andDarrell,T.Decaf: Adeepconvolutional
activation feature for generic visual recognition. CoRR,abs/1310.1531, 2013.
Everingham, M., Eslami, S.M. A., Van Gool, L., Williams,C., Winn, J., and Zisserman, A. The Pascal visual
object classes challenge: Aretrospective. IJCV,111(1):98–136, 2015.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An
incremental bayesian approach tested on 101 object categor ies. InIEEE CVPR Workshop of Generative
Model BasedVision , 2004.


incremental bayesian approach tested on 101 object categor ies. InIEEE CVPR Workshop of Generative
Model BasedVision , 2004.
Girshick, R. B., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection
and semantic segmentation. CoRR,abs/1311.2524v5, 2014. PublishedinProc.CVPR,2014.
Gkioxari, G.,Girshick, R.,and Malik, J. Actions and attrib utes from wholes and parts. CoRR,abs/1412.2604,
2014.
Glorot, X. andBengio, Y. Understanding the difﬁcultyof tra iningdeep feedforward neural networks. In Proc.
AISTATS,volume 9, pp. 249–256, 2010.
Goodfellow, I. J., Bulatov, Y., Ibarz, J., Arnoud, S., and Sh et, V. Multi-digit number recognition from street
view imagery usingdeep convolutional neural networks. In Proc.ICLR ,2014.
Grifﬁn, G., Holub, A., and Perona, P. Caltech-256 object cat egory dataset. Technical Report 7694, California
Institute of Technology, 2007.


Institute of Technology, 2007.
He, K., Zhang, X., Ren, S., and Sun, J. Spatial pyramid poolin g in deep convolutional networks for visual
recognition. CoRR,abs/1406.4729v2, 2014.
Hoai, M. Regularizedmax pooling forimage categorization. InProc. BMVC. ,2014.
Howard, A.G. Someimprovements ondeepconvolutional neura l networkbasedimageclassiﬁcation. In Proc.
ICLR,2014.
Jia, Y. Caffe: An open source convolutional architecture fo r fast feature embedding.
http://caffe.berkeleyvision.org/ ,2013.
Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignmen ts for generating image descriptions. CoRR,
abs/1412.2306, 2014.
Kiros, R., Salakhutdinov, R., and Zemel, R. S. Unifying visu al-semantic embeddings with multimodal neural
language models. CoRR,abs/1411.2539, 2014.
Krizhevsky, A. One weirdtrickfor parallelizingconvoluti onal neural networks. CoRR,abs/1404.5997, 2014.


Krizhevsky, A. One weirdtrickfor parallelizingconvoluti onal neural networks. CoRR,abs/1404.5997, 2014.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet cl assiﬁcation with deep convolutional neural networks. In NIPS,pp. 1106–1114, 2012.
LeCun,Y.,Boser, B.,Denker, J.S.,Henderson, D.,Howard, R .E.,Hubbard, W.,andJackel, L.D. Backpropagationapplied tohandwrittenzipcode recognition. Neural Computation , 1(4):541–551, 1989.
Lin,M., Chen, Q.,andYan, S. Networkinnetwork. In Proc.ICLR ,2014.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutiona l networks for semantic segmentation. CoRR,
abs/1411.4038, 2014.
Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning and Transferring Mid-Level Image Representations
using Convolutional Neural Networks. In Proc.CVPR ,2014.


using Convolutional Neural Networks. In Proc.CVPR ,2014.
Perronnin, F.,S´ anchez, J.,andMensink, T. Improving theF isherkernel forlarge-scale image classiﬁcation. In
Proc.ECCV ,2010.
Razavian, A.,Azizpour, H.,Sullivan, J.,andCarlsson,S. C NNFeaturesoff-the-shelf: anAstounding Baseline
for Recognition. CoRR,abs/1403.6382, 2014.
9
Publishedasa conferencepaperat ICLR2015
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A.,
Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet large sc ale visual recognition challenge. CoRR,
abs/1409.0575, 2014.
Sermanet,P.,Eigen, D.,Zhang, X.,Mathieu, M.,Fergus,R., andLeCun,Y. OverFeat: IntegratedRecognition,
Localizationand Detectionusing Convolutional Networks. InProc.ICLR ,2014.


Localizationand Detectionusing Convolutional Networks. InProc.ICLR ,2014.
Simonyan, K. and Zisserman, A. Two-stream convolutional ne tworks for action recognition in videos. CoRR,
abs/1406.2199, 2014. Published inProc.NIPS,2014.
Szegedy, C., Liu, W.,Jia, Y., Sermanet, P.,Reed, S.,Anguel ov, D.,Erhan, D., Vanhoucke, V., and Rabinovich,
A. Goingdeeper withconvolutions. CoRR,abs/1409.4842, 2014.
Wei, Y., Xia, W., Huang, J., Ni, B., Dong, J., Zhao, Y., and Yan , S. CNN: Single-label to multi-label. CoRR,
abs/1406.5726, 2014.
Zeiler, M. D. and Fergus, R. Visualizing and understanding c onvolutional networks. CoRR, abs/1311.2901,
2013. PublishedinProc. ECCV,2014.
A LOCALISATION
In the main bodyof the paper we have consideredthe classiﬁca tion task of the ILSVRC challenge,