conv1 and conv5 is larger ( 33) to compensate for
the increased spatial resolution. Compared to [10],
we use 5 convolutional layers as in the previous
architectures ([10] used 6), and less ﬁlters in conv5
(512 instead of 1024); we also incorporate an LRN
layer after conv1 ([10] did not use contrast normalisation).
3.2.1 CNN training
In general, our CNN training procedure follows that of [13], learning on ILSVRC-2012 using gradient descent with momentum. The hyperparameters are the same as used by [13]: momentum0:9; weight decay 5104; initial learning rate
102, which is decreased by a factor of 10, when
the validation error stop decreasing. The layers are
initialised from a Gaussian distribution with a zero
mean and variance equal to 102. We also employ
similar data augmentation in the form of random
crops, horizontal ﬂips, and RGB colour jittering.
Test time crop sampling is discussed in Sect. 3.3;
at training time, 224224 crops are sampled randomly, rather than deterministically. Thus, the only
notable difference to [13] is that the crops are taken
from the whole training image P256;P256,

Published in proceedings of BMVC 2014


notable difference to [13] is that the crops are taken
from the whole training image P256;P256,
rather than its 256256 centre. Training was performed on a single NVIDIA GTX Titan GPU and
the training time varied from 5 days for CNN-F to
3 weeks for CNN-S.
3.2.2 CNN ﬁne-tuning on the target dataset
In our experiments, we ﬁne-tuned CNN-S using
VOC-2007, VOC-2012, or Caltech-101 as the target
data. Fine-tuning was carried out using the same
framework (and the same data augmentation), as
we used for CNN training on ILSVRC. The last
5
Arch. conv1 conv2 conv3 conv4 conv5 full6 full7 full8
CNN-F64x11x11 256x5x5 256x3x3 256x3x3 256x3x3 4096 4096 1000
st. 4, pad 0 st. 1, pad 2 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- softLRN, x2 pool LRN, x2 pool - - x2 pool out out max

Return of the Devil in the Details: Delving Deep into Convolutional Nets

Summary:
The latest generation of Convolutional Neural Networks (CNN) have achieved
impressive results in challenging benchmarks on image recognition and object
detection, significantly raising the interest of the community in these
methods. Nevertheless, it is still unclear how different CNN methods compare
with each other and with previous state-of-the-art shallow representations such
as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper conducts
a rigorous evaluation of these new techniques, exploring different deep
architectures and comparing them on a common ground, identifying and disclosing
important implementation details. We identify several useful properties of
CNN-based representations, including the fact that the dimensionality of the
CNN output layer can be reduced significantly without having an adverse effect
on performance. We also identify aspects of deep and shallow methods that can
be successfully shared. In particular, we show that the data augmentation
techniques commonly applied to CNN-based methods can also be applied to shallow
methods, and result in an analogous performance boost. Source code and models
to reproduce the experiments in the paper is made publicly available.

Ken Chatfield
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

Published in proceedings of BMVC 2014


CNN-M96x7x7 256x5x5 512x3x3 512x3x3 512x3x3 4096 4096 1000
st. 2, pad 0 st. 2, pad 1 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- softLRN, x2 pool LRN, x2 pool - - x2 pool out out max
CNN-S96x7x7 256x5x5 512x3x3 512x3x3 512x3x3 4096 4096 1000
st. 2, pad 0 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 st. 1, pad 1 drop- drop- softLRN, x3 pool x2 pool - - x3 pool out out max
TABLE 1
CNN architectures. Each architecture contains 5 convolutional layers (conv 1–5) and three
fully-connected layers (full 1–3). The details of each of the convolutional layers are given in three sub-rows:
the ﬁrst speciﬁes the number of convolution ﬁlters and their receptive ﬁeld size as “num x size x size”; the
second indicates the convolution stride (“st.”) and spatial padding (“pad”); the third indicates if Local

Published in proceedings of BMVC 2014


second indicates the convolution stride (“st.”) and spatial padding (“pad”); the third indicates if Local
Response Normalisation (LRN) [13] is applied, and the max-pooling downsampling factor. For full 1–3, we
specify their dimensionality, which is the same for all three architectures. Full6 and full7 are regularised
using dropout [13], while the last layer acts as a multi-way soft-max classiﬁer. The activation function for all
weight layers (except for full8) is the REctiﬁcation Linear Unit (RELU) [13].
fully-connected layer (conv8) has output dimensionality equal to the number of classes, which
differs between datasets, so we initialised it from
a Gaussian distribution (as used for CNN training
above). Now we turn to dataset-speciﬁc ﬁne-tuning
details.
VOC-2007 and VOC-2012. Considering that PASCAL VOC is a multi-label dataset ( i.e. a single
image might have multiple labels), we replaced the
softmax regression loss with a more appropriate
loss function, for which we considered two options:
one-vs-rest classiﬁcation hinge loss (the same loss

Published in proceedings of BMVC 2014


softmax regression loss with a more appropriate
loss function, for which we considered two options:
one-vs-rest classiﬁcation hinge loss (the same loss
as used in the SVM experiments) and ranking hinge
loss. Both losses deﬁne constraints on the scores of
positive (Ipos) and negative ( Ineg) images for each
class:wc(Ipos)>1;wc(Ineg)<1 +for the
classiﬁcation loss, wc(Ipos)> w c(Ineg) + 1
for the ranking loss ( wcis thec-th row of the
last fully-connected layer, which can be seen as a
linear classiﬁer on deep features (I);is a slack
variable). Our ﬁne-tuned networks are denoted as
“CNN S TUNE-CLS” (for the classiﬁcation loss)
and “CNN S TUNE-RNK” (for the ranking loss).
In the case of both VOC datasets, the training
and validation subsets were combined to form a
single training set. Given the smaller size of the
training data when compared to ILSVRC-2012, we

Published in proceedings of BMVC 2014


and validation subsets were combined to form a
single training set. Given the smaller size of the
training data when compared to ILSVRC-2012, we
controlled for over-ﬁtting by using lower initial
learning rates for the ﬁne-tuned hidden layers.The learning rate schedule for the last layer /
hidden layers was: 102=104!103=104!
104=104!105=105.
Caltech-101 dataset contains a single class label
per image, so ﬁne-tuning was performed using the
softmax regression loss. Other settings (including
the learning rate schedule) were the same as used
for the VOC ﬁne-tuning experiments.
3.2.3 Low-dimensional CNN feature training
Our baseline networks (Table 1) have the same
dimensionality of the last hidden layer (full7): 4096.
This design choice is in accordance with the stateof-the-art architectures [13], [19], [10], and leads to
a 4096-D dimensional image representation, which
is already rather compact compared to IFV . We
further trained three modiﬁcations of the CNN-M
network, with lower dimensional full7 layers of:
2048, 1024, and 128 dimensions respectively. The

Published in proceedings of BMVC 2014


further trained three modiﬁcations of the CNN-M
network, with lower dimensional full7 layers of:
2048, 1024, and 128 dimensions respectively. The
networks were learnt on ILSVRC-2012. To speedup training, all layers aside from full7/full8 were
set to those of the CNN-M net and a lower initial
learning rate of 103was used. The initial learning
rate of full7/full8 was set to 102.
3.3 Data augmentation details
We explore three data augmentation strategies. The
ﬁrst strategy is to use no augmentation . In contrast to IFV , however, CNNs require images to
6
be transformed to a ﬁxed size ( 224224) even
when no augmentation is used. Hence the image is
downsized so that the smallest dimension is equal
to224 pixels and a 224224 crop is extracted
from the centre.2The second strategy is to use
ﬂip augmentation , mirroring images about the yaxis producing two samples from each image. The
third strategy, termed C+F augmentation , combines
cropping and ﬂipping. For CNN-based representations, the image is downsized so that the smallest
dimension is equal to 256 pixels. Then 224224

Published in proceedings of BMVC 2014


cropping and ﬂipping. For CNN-based representations, the image is downsized so that the smallest
dimension is equal to 256 pixels. Then 224224
crops are extracted from the four corners and the
centre of the image. Note that the crops are sampled
from the whole image, rather than its 256256
centre, as done by [13]. These crops are then ﬂipped
about they-axis, producing 10perturbed samples
per input image. In the case of the IFV encoding, the
same crops are extracted, but at the original image
resolution.
4 A NALYSIS
This section describes the experimental results,
comparing different features and data augmentation schemes. The results are given in Table 2 for
VOC-2007 and analysed next, starting from generally applicable methods such as augmentation and
then discussing the speciﬁcs of each scenario. We
then move onto other datasets and the state of the
art in Sect. 4.7.
4.1 Data augmentation
We experiment with no data augmentation (denoted Image Aug=– in Tab. 2), ﬂip augmentation ( Image Aug=F ), and C+F augmentation ( Image
Aug=C ). Augmented images are used as standalone samples ( f), or by fusing the corresponding

Published in proceedings of BMVC 2014


Aug=C ). Augmented images are used as standalone samples ( f), or by fusing the corresponding
descriptors using sum ( s) or max ( m) pooling or
stacking ( t). So for example Image Aug=(C) f s in
row [f] of Tab. 2 means that C+F augmentation is
used to generate additional samples in training ( f),
and is combined with sum-pooling in testing ( s).
Augmentation consistently improves performance by3%for both IFV ( e.g. [d] vs. [f]) and
CNN ( e.g. [o] vs. [p]). Using additional samples for
training and sum-pooling for testing works best
([p]) followed by sum-pooling [r], max pooling [q],
and stacking [s]. In terms of the choice of transformations, ﬂipping improves only marginally ([o] vs.
2. Extracting a 224224 centre crop from a 256256
image [13] resulted in worse performance.[u]), but using the more expensive C+F sampling
improves, as seen, by about 23%([o]vs. [p]). We
experimented with sampling more transformations,
taking a higher density of crops from the centre of
the image, but observed no beneﬁt.
4.2 Colour

Published in proceedings of BMVC 2014


experimented with sampling more transformations,
taking a higher density of crops from the centre of
the image, but observed no beneﬁt.
4.2 Colour
Colour information can be added and subtracted in
CNN and IFV . In IFV replacing SIFT with the colour
descriptors of [3] (denoted COL inMethod ) yields
signiﬁcantly worse performance ([j] vs. [h]). However, when SIFT and colour descriptors are combined by stacking the corresponding IFVs ( COL+ )
there is a small but signiﬁcant improvement of
around1%in the non-augmented case ( e.g. [h]
vs. [k]) but little impact in the augmented case
(e.g. [i] vs. [l]). For CNNs, retraining the network
after converting all the input images to grayscale
(denoted GSinMethods ) has a more signiﬁcant
impact, resulting in a performance drop of 3%
([w] vs. [p], [v] vs. [o]).
4.3 Scenario 1: Shallow representation (IFV)
The baseline IFV encoding using a spatial pyramid
[a] performs slightly better than the results [I] taken

Published in proceedings of BMVC 2014


4.3 Scenario 1: Shallow representation (IFV)
The baseline IFV encoding using a spatial pyramid
[a] performs slightly better than the results [I] taken
from Chatﬁeld et al. [12], primarily due to a larger
number of spatial scales being used during SIFT
feature extraction, and the resultant SIFT features
being square-rooted. Intra-normalisation , denoted as
INin the Method column of the table, improves the
performance by 1%(e.g. [c] vs. [d]). More interestingly, switching from spatial pooling (denoted
spm in the SPool column) to feature spatial augmentation ( SPool=(x,y) ) has either little effect on the
performance or results in a marginal increase ([a] vs.
[c], [b] vs. [d]), whilst resulting in a representation
which is over 10 smaller. We also experimented
with augmenting with scale in addition to position
as in [23] but observed no improvement. Finally,
we investigate pushing the parameters of the representation setting K= 512 (rows [h]-[l]). Increasing
the number of GMM centres in the model from
K= 256 to512 results in a further performance
increase (e.g. [h] vs. [d]), but at the expense of

Published in proceedings of BMVC 2014


K= 256 to512 results in a further performance
increase (e.g. [h] vs. [d]), but at the expense of
higher-dimensional codes (125K dimensional).
4.4 Scenario 2: Deep representation (CNN) with
pre-training
CNN-based methods consistently outperform the
shallow encodings, even after the improvements
7
Method SPool Image Aug. Dim mAP
(I) FK BL spm – 327K 61.69 79.0 67.4 51.9 70.9 30.8 72.2
(II) DECAF – (C) t t 327K 73.41 87.4 79.3 84.1 78.4 42.3 73.7
(a) FK spm – 327K 63.66 83.4 68.8 59.6 74.1 35.7 71.2
(b) FK IN spm – 327K 64.18 82.1 69.7 59.7 75.2 35.7 71.3
(c) FK (x,y) – 42K 63.51 83.2 69.4 60.6 73.9 36.3 68.6
(d) FK IN (x,y) – 42K 64.36 83.1 70.4 62.4 75.2 37.1 69.1

Published in proceedings of BMVC 2014


Published in proceedings of BMVC 2014


discussed above, by a large 10% mAP margin
([i]vs. [p]). Our small architecture CNN-F, which is
similar to DeCAF [7], performs signiﬁcantly better
than the latter ([II] vs. [s]), validating our implementation. Both medium CNN-M [m] and slow
CNN-S [p] outperform the fast CNN-F [m] by a
signiﬁcant 23% margin. Since the accuracy
of CNN-S and CNN-M is nearly the same, we
focus on the latter as it is simpler and marginally
(25%) faster. Remarkably, these good networks
work very well even with no augmentation [o].
Another advantage of CNNs compared to IFV is
the small dimensionality of the output features,although IFV can be compressed to an extent. We
explored retraining the CNNs such that the ﬁnal
layer was of a lower dimensionality, and reducing
from 4096 to 2048 actually resulted in a marginal
performance boost ([x] vs. [p]). What is surprising
is that we can reduce the output dimensionality
further to 1024D [y] and even 128D [z] with only
a drop of2%for codes that are 32smaller

Published in proceedings of BMVC 2014


(a) 86.8 85.6 59.9 72.0 73.4 81.4 88.6 80.5 92.1 60.6 77.3 66.4 89.3 73.3
(b) 87.7 88.4 61.2 76.9 76.6 84.9 89.1 82.9 92.4 61.9 80.9 68.7 91.5 75.1
(g) 91.5 91.9 64.1 76.3 74.9 89.7 92.2 86.9 95.2 60.7 82.9 68.0 95.5 74.4
TABLE 2
VOC 2007 results (continued from previous page)
applicable).
4.5 Scenario 3: Deep representation (CNN) with
pre-training and ﬁne-tuning
We ﬁne-tuned our CNN-S architecture on VOC2007 using the ranking hinge loss, and achieved
a signiﬁcant improvement: 2:7%([g]vs. [n]). This
demonstrates that in spite of the small amount
of VOC training data (5,011 images), ﬁne-tuning
is able to adjust the learnt deep representation to
better suit the dataset in question.4.6 Combinations
For the CNN-M 2048 representation [x], stacking

Published in proceedings of BMVC 2014


is able to adjust the learnt deep representation to
better suit the dataset in question.4.6 Combinations
For the CNN-M 2048 representation [x], stacking
deep and shallow representations to form a higherdimensional descriptor makes little difference ([x]
vs. [b]). For the weaker CNN-F it results in a small
boost of0:8%([m] vs. [a]).
4.7 Comparison with the state of the art
In Table 3 we report our results on ILSVRC-2012,
VOC-2007, VOC-2012, Caltech-101, and Caltech-256
datasets, and compare them to the state of the art.
First, we note that the ILSVRC error rates of our
CNN-F, CNN-M, and CNN-S networks are better
9
ILSVRC-2012 VOC-2007 VOC-2012 Caltech-101 Caltech-256
(top-5 error) (mAP) (mAP) (accuracy) (accuracy)
(a) FK IN 512 - 68.0 – – –
(b) CNN F 16.7 77.4 79.9 – –
(c) CNN M 13.7 79.9 82.5 87.15 0.80 77.030.46

Published in proceedings of BMVC 2014


(l) Wei et al. [17] - 81.5 ( 85.2*) 81.7 ( 90.3*) – –
(m) He et al. [29] 13.6 80.1 - 91.40.7 –
TABLE 3
Comparison with the state of the art on ILSVRC2012, VOC2007, VOC2012, Caltech-101, and
Caltech-256. Results marked with * were achieved using models pre-trained on the extended ILSVRC
datasets (1512 classes in [8], [16], 2000 classes in [17]). All other results were achieved using CNNs
pre-trained on ILSVRC-2012 (1000 classes).
than those reported by [13], [19], and [10] for the
related conﬁgurations. This validates our implementation, and the difference is likely to be due to
the sampling of image crops from the uncropped
image plane (instead of the centre). When using
our CNN features on other datasets, the relative
performance generally follows the same pattern
as on ILSVRC, where the nets are trained – the
CNN-F architecture exhibits the worst performance,
with CNN-M and CNN-S performing considerably
better.
Further ﬁne-tuning of CNN-S on the VOC

Published in proceedings of BMVC 2014


CNN-F architecture exhibits the worst performance,
with CNN-M and CNN-S performing considerably
better.
Further ﬁne-tuning of CNN-S on the VOC
datasets turns out to be beneﬁcial; on VOC-2012,
using the ranking loss is marginally better than
the classiﬁcation loss ([g] vs. [f]), which can be
explained by the ranking-based VOC evaluation
criterion. Fine-tuning on Caltech-101 also yields a
small improvement, but no gain is observed over
Caltech-256.
Our CNN-S net is competitive with recent CNNbased approaches [19], [9], [8], [16], [17], [29] and
on a number of datasets (VOC-2007, VOC-2012,
Caltech-101, Caltech-256) and sets the state of the
art on VOC-2007 and VOC-2012 across methods
pre-trained solely on ILSVRC-2012 dataset. While
the CNN-based methods of [16], [17] achieve better

Published in proceedings of BMVC 2014


pre-trained solely on ILSVRC-2012 dataset. While
the CNN-based methods of [16], [17] achieve better
performance on VOC (86.3% and 90.3% respectively), they were trained using extended ILSVRCdatasets, enriched with additional categories semantically close to the ones in VOC. Additionally, [17] used a signiﬁcantly more complex classiﬁcation pipeline, driven by bounding box proposals [30], pre-trained on ILSVRC-2013 detection
dataset. Their best reported result on VOC-2012
(90.3%) was achieved by the late fusion with a
complex hand-crafted method of [31]; without fusion, they get 84.2%. On Caltech-101, [29] achieves
the state of the art using spatial pyramid pooling
of conv5 layer features, while we used full7 layer
features consistently across all datasets (for full7
features, they report 87:08%).
In addition to achieving performance comparable
to the state of the art with a very simple approach
(but powerful CNN-based features), with the modiﬁcations outlined in the paper (primarily the use
of data augmentation similar to the CNN-based
methods) we are able to improve the performance

Published in proceedings of BMVC 2014


of data augmentation similar to the CNN-based
methods) we are able to improve the performance
of shallow IFV to 68.02% (Table 2, [i]).
4.8 Performance Evolution on VOC-2007
A comparative plot of the evolution in the performance of the methods evaluated in this paper,
along with a selection from our earlier review of
shallow methods [12] is presented in Fig. 1. Classiﬁcation accuracy over PASCAL VOC was 54.48%
mAP for the BoVW model in 2008, 61.7% for the
10
      BOW
32K
–FK-BL
327K
–
[I]FK
327K
–
[a]FK-IN
84K
f s
[i]DeCAF
4K
t t
[II]CNN-F
4K
f s
[m]CNN-M 2K
2K
f s
[y]CNN-S
4K (TN)
f s
[γ]

545658606264666870727476788082mAP 68.02
54.4861.6963.6673.4177.1580.132008 2010 2013 2014 ...
82.42
Method
Dim.
Aug.
Ref.

Published in proceedings of BMVC 2014


54.4861.6963.6673.4177.1580.132008 2010 2013 2014 ...
82.42
Method
Dim.
Aug.
Ref.
Fig. 1. Evolution of Performance on PASCAL VOC-2007 over the recent years. Please refer to Table 2
for details and references.
IFV in 2010 [12], and 73.41% for DeCAF [7] and
similar [8], [9] CNN-based methods introduced in
late 2013. Our best performing CNN-based method
(CNN-S with ﬁne-tuning) achieves 82.42%, comparable to the most recent state-of-the-art.
4.9 Timings and dimensionality
One of our best-performing CNN representations
CNN-M-2048 [x] is 42more compact than the
best performing IFV [i] (84K vs. 2K) and CNN-M
features are also 50faster to compute ( 120s
vs.2:4sper image with augmentation enabled,
over a single CPU core). Non-augmented CNN-M
features [o] take around 0:3sper image, compared
to0:4sfor CNN-S features and 0:13sfor CNNF features.
5 C ONCLUSION

Published in proceedings of BMVC 2014


to0:4sfor CNN-S features and 0:13sfor CNNF features.
5 C ONCLUSION
In this paper we presented a rigorous empirical evaluation of CNN-based methods for image
classiﬁcation, along with a comparison with more
traditional shallow feature encoding methods. We
have demonstrated that the performance of shallow representations can be signiﬁcantly improved
by adopting data augmentation, typically used indeep learning. In spite of this improvement, deep
architectures still outperform the shallow methods
by a large margin. We have shown that the performance of deep representations on the ILSVRC
dataset is a good indicator of their performance
on other datasets, and that ﬁne-tuning can further
improve on already very strong results achieved
using the combination of deep representations and
a linear SVM. Source code and CNN models to
reproduce the experiments presented in the paper
are available on the project website [32] in the
hope that it would provide common ground for
future comparisons, and good baselines for image
representation research.
ACKNOWLEDGEMENTS
This work was supported by the EPSRC and ERC
grant VisRec no. 228180. We gratefully acknowledge the support of NVIDIA Corporation with the
donation of the GPUs used for this research.
REFERENCES

Published in proceedings of BMVC 2014


grant VisRec no. 228180. We gratefully acknowledge the support of NVIDIA Corporation with the
donation of the GPUs used for this research.
REFERENCES
[1] G. Csurka, C. Bray, C. Dance, and L. Fan, “Visual categorization with bags of keypoints,” in Workshop on Statistical
Learning in Computer Vision, ECCV , 2004, pp. 1–22.
11
[2] J. Sivic and A. Zisserman, “Video Google: A text retrieval
approach to object matching in videos,” in Proc. ICCV ,
vol. 2, 2003, pp. 1470–1477.
[3] F. Perronnin, J. S ´anchez, and T. Mensink, “Improving the
Fisher kernel for large-scale image classiﬁcation,” in Proc.
ECCV , 2010.
[4] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural
Computation , vol. 1, no. 4, pp. 541–551, 1989.

Published in proceedings of BMVC 2014


Computation , vol. 1, no. 4, pp. 541–551, 1989.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei,
“Imagenet: A large-scale hierarchical image database,” in
Proc. CVPR , 2009.
[6] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman, “The PASCAL Visual Object Classes
(VOC) challenge,” IJCV , vol. 88, no. 2, pp. 303–338, 2010.
[7] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell, “Decaf: A deep convolutional
activation feature for generic visual recognition,” CoRR ,
vol. abs/1310.1531, 2013.
[8] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning
and Transferring Mid-Level Image Representations using
Convolutional Neural Networks,” in Proc. CVPR , 2014.

Published in proceedings of BMVC 2014


and Transferring Mid-Level Image Representations using
Convolutional Neural Networks,” in Proc. CVPR , 2014.
[9] A. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson,
“CNN Features off-the-shelf: an Astounding Baseline for
Recognition,” CoRR , vol. abs/1403.6382, 2014.
[10] P . Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y. LeCun, “OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks,” in
Proc. ICLR , 2014.
[11] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich
feature hierarchies for accurate object detection and semantic segmentation,” in Proc. CVPR , 2014.
[12] K. Chatﬁeld, V . Lempitsky, A. Vedaldi, and A. Zisserman,
“The devil is in the details: an evaluation of recent feature
encoding methods,” in Proc. BMVC. , 2011.

Published in proceedings of BMVC 2014


“The devil is in the details: an evaluation of recent feature
encoding methods,” in Proc. BMVC. , 2011.
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet
classiﬁcation with deep convolutional neural networks,”
inNIPS , 2012, pp. 1106–1114.
[14] M. Paulin, J. Revaud, Z. Harchaoui, F. Perronnin, and
C. Schmid, “Transformation Pursuit for Image Classiﬁcation,” in Proc. CVPR , 2014.
[15] F. Perronnin, Z. Akata, Z. Harchaoui, and C. Schmid,
“Towards good practice in large-scale learning for image
classiﬁcation,” in Proc. CVPR , 2012, pp. 3482–3489.
[16] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Weakly
supervised object recognition with convolutional neural
networks,” INRIA, Tech. Rep. HAL-01015140, 2014.

Published in proceedings of BMVC 2014


supervised object recognition with convolutional neural
networks,” INRIA, Tech. Rep. HAL-01015140, 2014.
[17] Y. Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y. Zhao, and
S. Yan, “CNN: Single-label to multi-label,” CoRR , vol.
abs/1406.5726, 2014.
[18] H. J ´egou, F. Perronnin, M. Douze, J. S ´anchez, P . P ´erez, and
C. Schmid, “Aggregating local images descriptors into
compact codes,” IEEE P AMI , 2012.
[19] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” CoRR , vol. abs/1311.2901,
2013.
[20] Y. Jia, “Caffe: An open source convolutional architecture
for fast feature embedding,” http://caffe.berkeleyvision.
org/, 2013.
[21] L. Fei-Fei, R. Fergus, and P . Perona, “Learning generative

Published in proceedings of BMVC 2014


org/, 2013.
[21] L. Fei-Fei, R. Fergus, and P . Perona, “Learning generative
visual models from few training examples: An incremental bayesian approach tested on 101 object categories,”
inIEEE CVPR Workshop of Generative Model Based Vision ,
2004.[22] G. Grifﬁn, A. Holub, and P . Perona, “Caltech-256 object
category dataset,” California Institute of Technology,
Tech. Rep. 7694, 2007. [Online]. Available: http://
authors.library.caltech.edu/7694
[23] J. S ´anchez, F. Perronnin, and T. Em ´ıdio de Campos,
“Modeling the spatial layout of images beyond spatial
pyramids,” Pattern Recognition Letters , vol. 33, no. 16, pp.
2216–2223, 2012.
[24] R. Arandjelovi ´c and A. Zisserman, “Three things everyone should know to improve object retrieval,” in Proc.
CVPR , 2012.
[25] A. Vedaldi and A. Zisserman, “Efﬁcient additive kernels
via explicit feature maps,” IEEE P AMI , 2011.

Published in proceedings of BMVC 2014


[25] A. Vedaldi and A. Zisserman, “Efﬁcient additive kernels
via explicit feature maps,” IEEE P AMI , 2011.
[26] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond Bags
of Features: Spatial Pyramid Matching for Recognizing
Natural Scene Categories,” in Proc. CVPR , 2006.
[27] R. Arandjelovi ´c and A. Zisserman, “All about VLAD,” in
Proc. CVPR , 2013.
[28] H. J ´egou, M. Douze, and C. Schmid, “On the burstiness
of visual elements,” in Proc. CVPR , Jun 2009.
[29] K. He, A. Zhang, S. Ren, and J. Sun, “Spatial pyramid
pooling in deep convolutional networks for visual recognition,” in Proc. ECCV , 2014.
[30] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P . H. S. Torr,
“BING: Binarized normed gradients for objectness estimation at 300fps,” in Proc. CVPR , 2014.

Published in proceedings of BMVC 2014


“BING: Binarized normed gradients for objectness estimation at 300fps,” in Proc. CVPR , 2014.
[31] S. Yan, J. Dong, Q. Chen, Z. Song, Y. Pan, W. Xia,
H. Zhongyang, Y. Hua, and S. Shen, “Generalized hierarchical matching for subcategory aware object classiﬁcation,” in The P ASCAL Visual Object Classes Challenge
Workshop , 2012.
[32] K. Chatﬁeld, K. Simonyan, A. Vedaldi, and A. Zisserman,
“Return of the devil in the details: delving deep into
convolutional nets webpage,” 2014. [Online]. Available:
http://www.robots.ox.ac.uk/vgg/research/deep eval

Published in proceedings of BMVC 2014


Learning Phrase Representations using RNN Encoder–Decoder
for Statistical Machine Translation
Kyunghyun Cho
Bart van Merri ¨enboer Caglar Gulcehre
Universit ´e de Montr ´eal
firstname.lastname@umontreal.caDzmitry Bahdanau
Jacobs University, Germany
d.bahdanau@jacobs-university.de
Fethi Bougares Holger Schwenk
Universit ´e du Maine, France
firstname.lastname@lium.univ-lemans.frYoshua Bengio
Universit ´e de Montr ´eal, CIFAR Senior Fellow
find.me@on.the.web
Abstract
In this paper, we propose a novel neural network model called RNN Encoder–
Decoder that consists of two recurrent
neural networks (RNN). One RNN encodes a sequence of symbols into a ﬁxedlength vector representation, and the other
decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly
trained to maximize the conditional probability of a target sequence given a source

Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Summary:
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.

Kyunghyun Cho
Bart van Merrienboer
Caglar Gulcehre
Dzmitry Bahdanau
Fethi Bougares
Holger Schwenk
Yoshua Bengio

EMNLP 2014


decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly
trained to maximize the conditional probability of a target sequence given a source
sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder–Decoder as an
additional feature in the existing log-linear
model. Qualitatively, we show that the
proposed model learns a semantically and
syntactically meaningful representation of
linguistic phrases.
1 Introduction
Deep neural networks have shown great success in
various applications such as objection recognition
(see, e.g., (Krizhevsky et al., 2012)) and speech
recognition (see, e.g., (Dahl et al., 2012)). Furthermore, many recent works showed that neural networks can be successfully used in a number of tasks in natural language processing (NLP).
These include, but are not limited to, language
modeling (Bengio et al., 2003), paraphrase detection (Socher et al., 2011) and word embedding extraction (Mikolov et al., 2013). In the ﬁeld of statistical machine translation (SMT), deep neural
networks have begun to show promising results.
(Schwenk, 2012) summarizes a successful usage
of feedforward neural networks in the framework

EMNLP 2014


networks have begun to show promising results.
(Schwenk, 2012) summarizes a successful usage
of feedforward neural networks in the framework
of phrase-based SMT system.Along this line of research on using neural networks for SMT, this paper focuses on a novel neural network architecture that can be used as a part
of the conventional phrase-based SMT system.
The proposed neural network architecture, which
we will refer to as an RNN Encoder–Decoder , consists of two recurrent neural networks (RNN) that
act as an encoder and a decoder pair. The encoder maps a variable-length source sequence to a
ﬁxed-length vector, and the decoder maps the vector representation back to a variable-length target
sequence. The two networks are trained jointly to
maximize the conditional probability of the target
sequence given a source sequence. Additionally,
we propose to use a rather sophisticated hidden
unit in order to improve both the memory capacity
and the ease of training.
The proposed RNN Encoder–Decoder with a
novel hidden unit is empirically evaluated on the
task of translating from English to French. We
train the model to learn the translation probability of an English phrase to a corresponding French
phrase. The model is then used as a part of a standard phrase-based SMT system by scoring each

EMNLP 2014


train the model to learn the translation probability of an English phrase to a corresponding French
phrase. The model is then used as a part of a standard phrase-based SMT system by scoring each
phrase pair in the phrase table. The empirical evaluation reveals that this approach of scoring phrase
pairs with an RNN Encoder–Decoder improves
the translation performance.
We qualitatively analyze the trained RNN
Encoder–Decoder by comparing its phrase scores
with those given by the existing translation model.
The qualitative analysis shows that the RNN
Encoder–Decoder is better at capturing the linguistic regularities in the phrase table, indirectly
explaining the quantitative improvements in the
overall translation performance. The further analysis of the model reveals that the RNN Encoder–
Decoder learns a continuous space representation
of a phrase that preserves both the semantic and
syntactic structure of the phrase.arXiv:1406.1078v3  [cs.CL]  3 Sep 2014
2 RNN Encoder–Decoder
2.1 Preliminary: Recurrent Neural Networks
A recurrent neural network (RNN) is a neural network that consists of a hidden state hand an
optional output ywhich operates on a variablelength sequence x= (x1;:::;x T). At each time
stept, the hidden state hhtiof the RNN is updated
by

EMNLP 2014


stept, the hidden state hhtiof the RNN is updated
by
hhti=f
hht1i;xt
; (1)
wherefis a non-linear activation function.fmay be as simple as an elementwise logistic sigmoid function and as complex as a long short-term memory (LSTM)
unit (Hochreiter and Schmidhuber, 1997).
An RNN can learn a probability distribution
over a sequence by being trained to predict the
next symbol in a sequence. In that case, the output
at each timestep tis the conditional distribution
p(xtjxt1;:::;x 1). For example, a multinomial
distribution ( 1-of-Kcoding) can be output using a
softmax activation function
p(xt;j= 1jxt1;:::;x 1) =exp
wjhhti
PK
j0=1exp
wj0hhti;
(2)
for all possible symbols j= 1;:::;K , where wj
are the rows of a weight matrix W. By combining
these probabilities, we can compute the probability of the sequence xusing
p(x) =TY

EMNLP 2014


are the rows of a weight matrix W. By combining
these probabilities, we can compute the probability of the sequence xusing
p(x) =TY
t=1p(xtjxt1;:::;x 1): (3)
From this learned distribution, it is straightforward to sample a new sequence by iteratively sampling a symbol at each time step.
2.2 RNN Encoder–Decoder
In this paper, we propose a novel neural network
architecture that learns to encode a variable-length
sequence into a ﬁxed-length vector representation
and to decode a given ﬁxed-length vector representation back into a variable-length sequence.
From a probabilistic perspective, this new model
is a general method to learn the conditional distribution over a variable-length sequence conditioned on yet another variable-length sequence,
e.g.p(y1;:::;y T0jx1;:::;x T), where one
x1x2 xTyT' y2 y1
cDecoder
EncoderFigure 1: An illustration of the proposed RNN
Encoder–Decoder.
should note that the input and output sequence
lengthsTandT0may differ.
The encoder is an RNN that reads each symbol
of an input sequence xsequentially. As it reads

EMNLP 2014


lengthsTandT0may differ.
The encoder is an RNN that reads each symbol
of an input sequence xsequentially. As it reads
each symbol, the hidden state of the RNN changes
according to Eq. (1). After reading the end of
the sequence (marked by an end-of-sequence symbol), the hidden state of the RNN is a summary c
of the whole input sequence.
The decoder of the proposed model is another
RNN which is trained to generate the output sequence by predicting the next symbol ytgiven the
hidden state hhti. However, unlike the RNN described in Sec. 2.1, both ytandhhtiare also conditioned onyt1and on the summary cof the input
sequence. Hence, the hidden state of the decoder
at timetis computed by,
hhti=f
hht1i;yt1;c
;
and similarly, the conditional distribution of the
next symbol is
P(ytjyt1;yt2;:::;y 1;c) =g
hhti;yt1;c
:
for given activation functions fandg(the latter
must produce valid probabilities, e.g. with a softmax).
See Fig. 1 for a graphical depiction of the proposed model architecture.

EMNLP 2014


for given activation functions fandg(the latter
must produce valid probabilities, e.g. with a softmax).
See Fig. 1 for a graphical depiction of the proposed model architecture.
The two components of the proposed RNN
Encoder–Decoder are jointly trained to maximize
the conditional log-likelihood
max
1
NNX
n=1logp(ynjxn); (4)
where is the set of the model parameters and
each (xn;yn)is an (input sequence, output sequence) pair from the training set. In our case,
as the output of the decoder, starting from the input, is differentiable, we can use a gradient-based
algorithm to estimate the model parameters.
Once the RNN Encoder–Decoder is trained, the
model can be used in two ways. One way is to use
the model to generate a target sequence given an
input sequence. On the other hand, the model can
be used to score a given pair of input and output
sequences, where the score is simply a probability
p(yjx)from Eqs. (3) and (4).
2.3 Hidden Unit that Adaptively Remembers
and Forgets
In addition to a novel model architecture, we also
propose a new type of hidden unit ( fin Eq. (1))

EMNLP 2014


and Forgets
In addition to a novel model architecture, we also
propose a new type of hidden unit ( fin Eq. (1))
that has been motivated by the LSTM unit but is
much simpler to compute and implement.1Fig. 2
shows the graphical depiction of the proposed hidden unit.
Let us describe how the activation of the j-th
hidden unit is computed. First, the reset gaterjis
computed by
rj=
[Wrx]j+
Urhht1i
j
; (5)
whereis the logistic sigmoid function, and [:]j
denotes the j-th element of a vector. xandht1
are the input and the previous hidden state, respectively. WrandUrare weight matrices which are
learned.
Similarly, the update gatezjis computed by
zj=
[Wzx]j+
Uzhht1i
j
: (6)
The actual activation of the proposed unit hjis
then computed by
hhti
j=zjhht1i
j + (1zj)~hhti
j; (7)
where
~hhti
j=
[Wx]j+
U

EMNLP 2014


j; (7)
where
~hhti
j=
[Wx]j+
U
rhht1i
j
:(8)
In this formulation, when the reset gate is close
to 0, the hidden state is forced to ignore the previous hidden state and reset with the current input
1The LSTM unit, which has shown impressive results in
several applications such as speech recognition, has a memory cell and four gating units that adaptively control the information ﬂow inside the unit, compared to only two gating
units in the proposed hidden unit. For details on LSTM networks, see, e.g., (Graves, 2012).
z
rh h~xFigure 2: An illustration of the proposed hidden
activation function. The update gate zselects
whether the hidden state is to be updated with
a new hidden state ~h. The reset gate rdecides
whether the previous hidden state is ignored. See
Eqs. (5)–(8) for the detailed equations of r,z,h
and~h.
only. This effectively allows the hidden state to
drop any information that is found to be irrelevant
later in the future, thus, allowing a more compact
representation.
On the other hand, the update gate controls how
much information from the previous hidden state

EMNLP 2014


later in the future, thus, allowing a more compact
representation.
On the other hand, the update gate controls how
much information from the previous hidden state
will carry over to the current hidden state. This
acts similarly to the memory cell in the LSTM
network and helps the RNN to remember longterm information. Furthermore, this may be considered an adaptive variant of a leaky-integration
unit (Bengio et al., 2013).
As each hidden unit has separate reset and update gates, each hidden unit will learn to capture
dependencies over different time scales. Those
units that learn to capture short-term dependencies
will tend to have reset gates that are frequently active, but those that capture longer-term dependencies will have update gates that are mostly active.
In our preliminary experiments, we found that
it is crucial to use this new unit with gating units.
We were not able to get meaningful result with an
oft-used tanh unit without any gating.
3 Statistical Machine Translation
In a commonly used statistical machine translation
system (SMT), the goal of the system (decoder,
speciﬁcally) is to ﬁnd a translation fgiven a source
sentence e, which maximizes
p(fje)/p(ejf)p(f);
where the ﬁrst term at the right hand side is called

EMNLP 2014


sentence e, which maximizes
p(fje)/p(ejf)p(f);
where the ﬁrst term at the right hand side is called
translation model and the latter language model
(see, e.g., (Koehn, 2005)). In practice, however,
most SMT systems model logp(fje)as a loglinear model with additional features and corresponding weights:
logp(fje) =NX
n=1wnfn(f;e) + logZ(e);(9)
wherefnandwnare then-th feature and weight,
respectively. Z(e)is a normalization constant that
does not depend on the weights. The weights are
often optimized to maximize the BLEU score on a
development set.
In the phrase-based SMT framework
introduced in (Koehn et al., 2003) and
(Marcu and Wong, 2002), the translation model
logp(ejf)is factorized into the translation
probabilities of matching phrases in the source
and target sentences.2These probabilities are
once again considered additional features in the
log-linear model (see Eq. (9)) and are weighted
accordingly to maximize the BLEU score.
Since the neural net language model was proposed in (Bengio et al., 2003), neural networks

EMNLP 2014


accordingly to maximize the BLEU score.
Since the neural net language model was proposed in (Bengio et al., 2003), neural networks
have been used widely in SMT systems. In
many cases, neural networks have been used to
rescore translation hypotheses ( n-best lists) (see,
e.g., (Schwenk et al., 2006)). Recently, however,
there has been interest in training neural networks
to score the translated sentence (or phrase pairs)
using a representation of the source sentence as
an additional input. See, e.g., (Schwenk, 2012),
(Son et al., 2012) and (Zou et al., 2013).
3.1 Scoring Phrase Pairs with RNN
Encoder–Decoder
Here we propose to train the RNN Encoder–
Decoder (see Sec. 2.2) on a table of phrase pairs
and use its scores as additional features in the loglinear model in Eq. (9) when tuning the SMT decoder.
When we train the RNN Encoder–Decoder, we
ignore the (normalized) frequencies of each phrase
pair in the original corpora. This measure was
taken in order (1) to reduce the computational expense of randomly selecting phrase pairs from a

EMNLP 2014


pair in the original corpora. This measure was
taken in order (1) to reduce the computational expense of randomly selecting phrase pairs from a
large phrase table according to the normalized frequencies and (2) to ensure that the RNN Encoder–
Decoder does not simply learn to rank the phrase
pairs according to their numbers of occurrences.
One underlying reason for this choice was that the
existing translation probability in the phrase table already reﬂects the frequencies of the phrase
2Without loss of generality, from here on, we refer to
p(ejf)for each phrase pair as a translation model as wellpairs in the original corpus. With a ﬁxed capacity
of the RNN Encoder–Decoder, we try to ensure
that most of the capacity of the model is focused
toward learning linguistic regularities, i.e., distinguishing between plausible and implausible translations, or learning the “manifold” (region of probability concentration) of plausible translations.
Once the RNN Encoder–Decoder is trained, we
add a new score for each phrase pair to the existing phrase table. This allows the new scores to enter into the existing tuning algorithm with minimal
additional overhead in computation.
As Schwenk pointed out in (Schwenk, 2012),
it is possible to completely replace the existing

EMNLP 2014


additional overhead in computation.
As Schwenk pointed out in (Schwenk, 2012),
it is possible to completely replace the existing
phrase table with the proposed RNN Encoder–
Decoder. In that case, for a given source phrase,
the RNN Encoder–Decoder will need to generate
a list of (good) target phrases. This requires, however, an expensive sampling procedure to be performed repeatedly. In this paper, thus, we only
consider rescoring the phrase pairs in the phrase
table.
3.2 Related Approaches: Neural Networks in
Machine Translation
Before presenting the empirical results, we discuss
a number of recent works that have proposed to
use neural networks in the context of SMT.
Schwenk in (Schwenk, 2012) proposed a similar approach of scoring phrase pairs. Instead of the
RNN-based neural network, he used a feedforward
neural network that has ﬁxed-size inputs (7 words
in his case, with zero-padding for shorter phrases)
and ﬁxed-size outputs (7 words in the target language). When it is used speciﬁcally for scoring
phrases for the SMT system, the maximum phrase
length is often chosen to be small. However, as the
length of phrases increases or as we apply neural

EMNLP 2014


phrases for the SMT system, the maximum phrase
length is often chosen to be small. However, as the
length of phrases increases or as we apply neural
networks to other variable-length sequence data,
it is important that the neural network can handle variable-length input and output. The proposed RNN Encoder–Decoder is well-suited for
these applications.
Similar to (Schwenk, 2012), Devlin et al.
(Devlin et al., 2014) proposed to use a feedforward neural network to model a translation model,
however, by predicting one word in a target phrase
at a time. They reported an impressive improvement, but their approach still requires the maximum length of the input phrase (or context words)
to be ﬁxed a priori.
Although it is not exactly a neural network they
train, the authors of (Zou et al., 2013) proposed
to learn a bilingual embedding of words/phrases.
They use the learned embedding to compute the
distance between a pair of phrases which is used
as an additional score of the phrase pair in an SMT
system.
In (Chandar et al., 2014), a feedforward neural
network was trained to learn a mapping from a
bag-of-words representation of an input phrase to
an output phrase. This is closely related to both the

EMNLP 2014


network was trained to learn a mapping from a
bag-of-words representation of an input phrase to
an output phrase. This is closely related to both the
proposed RNN Encoder–Decoder and the model
proposed in (Schwenk, 2012), except that their input representation of a phrase is a bag-of-words.
A similar approach of using bag-of-words representations was proposed in (Gao et al., 2013) as
well. Earlier, a similar encoder–decoder model using two recursive neural networks was proposed
in (Socher et al., 2011), but their model was restricted to a monolingual setting, i.e. the model
reconstructs an input sentence. More recently, another encoder–decoder model using an RNN was
proposed in (Auli et al., 2013), where the decoder is conditioned on a representation of either
a source sentence or a source context.
One important difference between the proposed RNN Encoder–Decoder and the approaches
in (Zou et al., 2013) and (Chandar et al., 2014) is
that the order of the words in source and target phrases is taken into account. The RNN
Encoder–Decoder naturally distinguishes between
sequences that have the same words but in a different order, whereas the aforementioned approaches
effectively ignore order information.
The closest approach related to the proposed

EMNLP 2014


sequences that have the same words but in a different order, whereas the aforementioned approaches
effectively ignore order information.
The closest approach related to the proposed
RNN Encoder–Decoder is the Recurrent Continuous Translation Model (Model 2) proposed in
(Kalchbrenner and Blunsom, 2013). In their paper, they proposed a similar model that consists
of an encoder and decoder. The difference with
our model is that they used a convolutional n-gram
model (CGM) for the encoder and the hybrid of
an inverse CGM and a recurrent neural network
for the decoder. They, however, evaluated their
model on rescoring the n-best list proposed by the
conventional SMT system and computing the perplexity of the gold standard translations.
4 Experiments
We evaluate our approach on the English/French
translation task of the WMT’14 workshop.4.1 Data and Baseline System
Large amounts of resources are available to build
an English/French SMT system in the framework
of the WMT’14 translation task. The bilingual
corpora include Europarl (61M words), news commentary (5.5M), UN (421M), and two crawled
corpora of 90M and 780M words respectively.
The last two corpora are quite noisy. To train

EMNLP 2014


corpora of 90M and 780M words respectively.
The last two corpora are quite noisy. To train
the French language model, about 712M words of
crawled newspaper material is available in addition to the target side of the bitexts. All the word
counts refer to French words after tokenization.
It is commonly acknowledged that training statistical models on the concatenation of all this
data does not necessarily lead to optimal performance, and results in extremely large models which are difﬁcult to handle. Instead, one
should focus on the most relevant subset of the
data for a given task. We have done so by
applying the data selection method proposed in
(Moore and Lewis, 2010), and its extension to bitexts (Axelrod et al., 2011). By these means we
selected a subset of 418M words out of more
than 2G words for language modeling and a
subset of 348M out of 850M words for training the RNN Encoder–Decoder. We used the
test set newstest2012 and 2013 for data
selection and weight tuning with MERT, and
newstest2014 as our test set. Each set has
more than 70 thousand words and a single reference translation.
For training the neural networks, including the
proposed RNN Encoder–Decoder, we limited the
source and target vocabulary to the most frequent

EMNLP 2014


For training the neural networks, including the
proposed RNN Encoder–Decoder, we limited the
source and target vocabulary to the most frequent
15,000 words for both English and French. This
covers approximately 93% of the dataset. All the
out-of-vocabulary words were mapped to a special
token ( [UNK ]).
The baseline phrase-based SMT system was
built using Moses with default settings. This system achieves a BLEU score of 30.64 and 33.3 on
the development and test sets, respectively (see Table 1).
4.1.1 RNN Encoder–Decoder
The RNN Encoder–Decoder used in the experiment had 1000 hidden units with the proposed
gates at the encoder and at the decoder. The input matrix between each input symbol xhtiand the
hidden unit is approximated with two lower-rank
matrices, and the output matrix is approximated
ModelsBLEU
dev test
Baseline 30.64 33.30
RNN 31.20 33.87
CSLM + RNN 31.48 34.64
CSLM + RNN + WP 31.50 34.54
Table 1: BLEU scores computed on the development and test sets using different combinations of
approaches. WP denotes a word penalty , where
we penalizes the number of unknown words to

EMNLP 2014


Table 1: BLEU scores computed on the development and test sets using different combinations of
approaches. WP denotes a word penalty , where
we penalizes the number of unknown words to
neural networks.
similarly. We used rank-100 matrices, equivalent
to learning an embedding of dimension 100 for
each word. The activation function used for ~hin
Eq. (8) is a hyperbolic tangent function. The computation from the hidden state in the decoder to
the output is implemented as a deep neural network (Pascanu et al., 2014) with a single intermediate layer having 500 maxout units each pooling
2 inputs (Goodfellow et al., 2013).
All the weight parameters in the RNN Encoder–
Decoder were initialized by sampling from an
isotropic zero-mean (white) Gaussian distribution
with its standard deviation ﬁxed to 0:01, except
for the recurrent weight parameters. For the recurrent weight matrices, we ﬁrst sampled from a
white Gaussian distribution and used its left singular vectors matrix, following (Saxe et al., 2014).
We used Adadelta and stochastic gradient
descent to train the RNN Encoder–Decoder
with hyperparameters = 106and=
0:95(Zeiler, 2012). At each update, we used 64

EMNLP 2014


with hyperparameters = 106and=
0:95(Zeiler, 2012). At each update, we used 64
randomly selected phrase pairs from a phrase table (which was created from 348M words). The
model was trained for approximately three days.
Details of the architecture used in the experiments are explained in more depth in the supplementary material.
4.1.2 Neural Language Model
In order to assess the effectiveness of scoring
phrase pairs with the proposed RNN Encoder–
Decoder, we also tried a more traditional approach
of using a neural network for learning a target
language model (CSLM) (Schwenk, 2007). Especially, the comparison between the SMT system
using CSLM and that using the proposed approach
of phrase scoring by RNN Encoder–Decoder will
clarify whether the contributions from multiple
neural networks in different parts of the SMT sys-tem add up or are redundant.
We trained the CSLM model on 7-grams
from the target corpus. Each input word
was projected into the embedding space R512,
and they were concatenated to form a 3072dimensional vector. The concatenated vector was
fed through two rectiﬁed layers (of size 1536 and
1024) (Glorot et al., 2011). The output layer was

EMNLP 2014


fed through two rectiﬁed layers (of size 1536 and
1024) (Glorot et al., 2011). The output layer was
a simple softmax layer (see Eq. (2)). All the
weight parameters were initialized uniformly between0:01and0:01, and the model was trained
until the validation perplexity did not improve for
10 epochs. After training, the language model
achieved a perplexity of 45.80. The validation set
was a random selection of 0.1% of the corpus. The
model was used to score partial translations during the decoding process, which generally leads to
higher gains in BLEU score than n-best list rescoring (Vaswani et al., 2013).
To address the computational complexity of
using a CSLM in the decoder a buffer was
used to aggregate n-grams during the stacksearch performed by the decoder. Only when
the buffer is full, or a stack is about to
be pruned, the n-grams are scored by the
CSLM. This allows us to perform fast matrixmatrix multiplication on GPU using Theano
(Bergstra et al., 2010; Bastien et al., 2012).
−60 −50 −40 −30 −20 −10 0−14−12−10−8−6−4−20

EMNLP 2014


−60 −50 −40 −30 −20 −10 0−14−12−10−8−6−4−20
RNN Scores (log)TM Scores (log)
Figure 3: The visualization of phrase pairs according to their scores (log-probabilities) by the RNN
Encoder–Decoder and the translation model.
4.2 Quantitative Analysis
We tried the following combinations:
1. Baseline conﬁguration
2. Baseline + RNN
3. Baseline + CSLM + RNN
4. Baseline + CSLM + RNN + Word penalty
Source Translation Model RNN Encoder–Decoder
at the end of the [a la ﬁn de la] [ ´r la ﬁn des ann ´ees] [ ˆetre supprim ´es`a la ﬁn de la][`a la ﬁn du] [ `a la ﬁn des] [ `a la ﬁn de la]
for the ﬁrst time [rc
pour la premi r¨ere fois] [ ´et´e donn ´es pour
la premi `ere fois] [ ´et´e comm ´emor ´ee pour la

EMNLP 2014


la premi `ere fois] [ ´et´e comm ´emor ´ee pour la
premi `ere fois][pour la premi `ere fois] [pour la premi `ere fois ,]
[pour la premi `ere fois que]
in the United States
and[?aux?tats-Unis et] [ ´et´e ouvertes aux ´EtatsUnis et] [ ´et´e constat ´ees aux ´Etats-Unis et][aux Etats-Unis et] [des Etats-Unis et] [des
´Etats-Unis et]
, as well as [?s , qu’] [ ?s , ainsi que] [ ?re aussi bien que] [, ainsi qu’] [, ainsi que] [, ainsi que les]
one of the most [?t?l’ un des plus] [ ?l’ un des plus] [ ˆetre retenue
comme un de ses plus][l’ un des] [le] [un des]
(a) Long, frequent source phrases
Source Translation Model RNN Encoder–Decoder

EMNLP 2014


(a) Long, frequent source phrases
Source Translation Model RNN Encoder–Decoder
, Minister of Communications and Transport[Secr ´etaire aux communications et aux transports :] [Secr ´etaire aux communications et aux
transports][Secr ´etaire aux communications et aux transports] [Secr ´etaire aux communications et aux
transports :]
did not comply with
the[vestimentaire , ne correspondaient pas `a des]
[susmentionn ´ee n’ ´etait pas conforme aux]
[pr´esent ´ees n’ ´etaient pas conformes `a la][n’ ont pas respect ´e les] [n’ ´etait pas conforme
aux] [n’ ont pas respect ´e la]
parts of the world . [c
gions du monde .] [r ´egions du monde consid´er´ees .] [r ´egion du monde consid ´er´ee .][parties du monde .] [les parties du monde .]
[des parties du monde .]
the past few days . [le petit texte .] [cours des tout derniers jours .]

EMNLP 2014


[des parties du monde .]
the past few days . [le petit texte .] [cours des tout derniers jours .]
[les tout derniers jours .][ces derniers jours .] [les derniers jours .] [cours
des derniers jours .]
on Friday and Saturday[vendredi et samedi `a la] [vendredi et samedi `a]
[se d ´eroulera vendredi et samedi ,][le vendredi et le samedi] [le vendredi et samedi]
[vendredi et samedi]
(b) Long, rare source phrases
Table 2: The top scoring target phrases for a small set of source phrases according to the translation
model (direct translation probability) and by the RNN Encoder–Decoder. Source phrases were randomly
selected from phrases with 4 or more words. ?denotes an incomplete (partial) character. ris a Cyrillic
letter ghe.
The results are presented in Table 1. As expected, adding features computed by neural networks consistently improves the performance over
the baseline performance.
The best performance was achieved when we
used both CSLM and the phrase scores from the
RNN Encoder–Decoder. This suggests that the

EMNLP 2014


the baseline performance.
The best performance was achieved when we
used both CSLM and the phrase scores from the
RNN Encoder–Decoder. This suggests that the
contributions of the CSLM and the RNN Encoder–
Decoder are not too correlated and that one can
expect better results by improving each method independently. Furthermore, we tried penalizing the
number of words that are unknown to the neural
networks (i.e. words which are not in the shortlist). We do so by simply adding the number of
unknown words as an additional feature the loglinear model in Eq. (9).3However, in this case we
3To understand the effect of the penalty, consider the set
of all words in the 15,000 large shortlist, SL. All words xi=2
SL are replaced by a special token [UNK]before being scored
by the neural networks. Hence, the conditional probability of
anyxi
t=2SL is actually given by the model as
p(xt= [UNK]jx<t) =p(xt=2SLjx<t)
=X
xj
t=2SLp
xj
tjx<t
p
xi
tjx<t
;

EMNLP 2014


=X
xj
t=2SLp
xj
tjx<t
p
xi
tjx<t
;
where x<tis a shorthand notation for xt1; : : : ; x 1.were not able to achieve better performance on the
test set, but only on the development set.
4.3 Qualitative Analysis
In order to understand where the performance improvement comes from, we analyze the phrase pair
scores computed by the RNN Encoder–Decoder
against the corresponding p(fje)from the translation model. Since the existing translation model
relies solely on the statistics of the phrase pairs in
the corpus, we expect its scores to be better estimated for the frequent phrases but badly estimated
for rare phrases. Also, as we mentioned earlier
in Sec. 3.1, we further expect the RNN Encoder–
Decoder which was trained without any frequency
information to score the phrase pairs based rather
on the linguistic regularities than on the statistics
of their occurrences in the corpus.
We focus on those pairs whose source phrase is
long (more than 3 words per source phrase) and
As a result, the probability of words not in the shortlist is
always overestimated. It is possible to address this issue by
backing off to an existing model that contain non-shortlisted

EMNLP 2014


always overestimated. It is possible to address this issue by
backing off to an existing model that contain non-shortlisted
words (see (Schwenk, 2007)) In this paper, however, we opt
for introducing a word penalty instead, which counteracts the
word probability overestimation.
Source Samples from RNN Encoder–Decoder
at the end of the [`a la ﬁn de la] (11)
for the ﬁrst time [pour la premi `ere fois] (24) [pour la premi `ere fois que] (2)
in the United States and [aux ´Etats-Unis et] (6) [dans les ´Etats-Unis et] (4)
, as well as [, ainsi que] [,] [ainsi que] [, ainsi qu’] [et UNK]
one of the most [l’ un des plus] (9) [l’ un des] (5) [l’ une des plus] (2)
(a) Long, frequent source phrases
Source Samples from RNN Encoder–Decoder
, Minister of Communications and Transport[ , ministre des communications et le transport] ( 13)

EMNLP 2014


Source Samples from RNN Encoder–Decoder
, Minister of Communications and Transport[ , ministre des communications et le transport] ( 13)
did not comply with the [n’ tait pas conforme aux] [n’ a pas respect l’] ( 2) [n’ a pas respect la] ( 3)
parts of the world . [arts du monde .] (11) [des arts du monde .] ( 7)
the past few days . [quelques jours .] (5) [les derniers jours .] ( 5) [ces derniers jours .] ( 2)
on Friday and Saturday [vendredi et samedi] ( 5) [le vendredi et samedi] ( 7) [le vendredi et le samedi] ( 4)
(b) Long, rare source phrases
Table 3: Samples generated from the RNN Encoder–Decoder for each source phrase used in Table 2. We
show the top-5 target phrases out of 50 samples. They are sorted by the RNN Encoder–Decoder scores.
Figure 4: 2–D embedding of the learned word representation. The left one shows the full embedding

EMNLP 2014


Figure 4: 2–D embedding of the learned word representation. The left one shows the full embedding
space, while the right one shows a zoomed-in view of one region (color–coded). For more plots, see the
supplementary material.
frequent . For each such source phrase, we look
at the target phrases that have been scored high
either by the translation probability p(fje)or
by the RNN Encoder–Decoder. Similarly, we perform the same procedure with those pairs whose
source phrase is long butrare in the corpus.
Table 2 lists the top- 3target phrases per source
phrase favored either by the translation model
or by the RNN Encoder–Decoder. The source
phrases were randomly chosen among long ones
having more than 4 or 5 words.
In most cases, the choices of the target phrases
by the RNN Encoder–Decoder are closer to actual or literal translations. We can observe that the
RNN Encoder–Decoder prefers shorter phrases in
general.
Interestingly, many phrase pairs were scored
similarly by both the translation model and the
RNN Encoder–Decoder, but there were as manyother phrase pairs that were scored radically different (see Fig. 3). This could arise from the
proposed approach of training the RNN Encoder–

EMNLP 2014


proposed approach of training the RNN Encoder–
Decoder on a set of unique phrase pairs, discouraging the RNN Encoder–Decoder from learning
simply the frequencies of the phrase pairs from the
corpus, as explained earlier.
Furthermore, in Table 3, we show for each of
the source phrases in Table 2, the generated samples from the RNN Encoder–Decoder. For each
source phrase, we generated 50 samples and show
the top-ﬁve phrases accordingly to their scores.
We can see that the RNN Encoder–Decoder is
able to propose well-formed target phrases without looking at the actual phrase table. Importantly,
the generated phrases do not overlap completely
with the target phrases from the phrase table. This
encourages us to further investigate the possibility
of replacing the whole or a part of the phrase table
Figure 5: 2–D embedding of the learned phrase representation. The top left one shows the full representation space (5000 randomly selected points), while the other three ﬁgures show the zoomed-in view of
speciﬁc regions (color–coded).
with the proposed RNN Encoder–Decoder in the
future.
4.4 Word and Phrase Representations
Since the proposed RNN Encoder–Decoder is not

EMNLP 2014


with the proposed RNN Encoder–Decoder in the
future.
4.4 Word and Phrase Representations
Since the proposed RNN Encoder–Decoder is not
speciﬁcally designed only for the task of machine
translation, here we brieﬂy look at the properties
of the trained model.
It has been known for some time that
continuous space language models using
neural networks are able to learn semantically meaningful embeddings (See, e.g.,
(Bengio et al., 2003; Mikolov et al., 2013)). Since
the proposed RNN Encoder–Decoder also projects
to and maps back from a sequence of words into
a continuous space vector, we expect to see a
similar property with the proposed model as well.
The left plot in Fig. 4 shows the 2–D embedding
of the words using the word embedding matrix
learned by the RNN Encoder–Decoder. The projection was done by the recently proposed BarnesHut-SNE (van der Maaten, 2013). We can clearly
see that semantically similar words are clusteredwith each other (see the zoomed-in plots in Fig. 4).
The proposed RNN Encoder–Decoder naturally
generates a continuous-space representation of a
phrase. The representation ( cin Fig. 1) in this

EMNLP 2014


The proposed RNN Encoder–Decoder naturally
generates a continuous-space representation of a
phrase. The representation ( cin Fig. 1) in this
case is a 1000-dimensional vector. Similarly to the
word representations, we visualize the representations of the phrases that consists of four or more
words using the Barnes-Hut-SNE in Fig. 5.
From the visualization, it is clear that the RNN
Encoder–Decoder captures both semantic and syntactic structures of the phrases. For instance, in
the bottom-left plot, most of the phrases are about
the duration of time, while those phrases that are
syntactically similar are clustered together. The
bottom-right plot shows the cluster of phrases that
are semantically similar (countries or regions). On
the other hand, the top-right plot shows the phrases
that are syntactically similar.
5 Conclusion
In this paper, we proposed a new neural network
architecture, called an RNN Encoder–Decoder
that is able to learn the mapping from a sequence
of an arbitrary length to another sequence, possibly from a different set, of an arbitrary length. The
proposed RNN Encoder–Decoder is able to either
score a pair of sequences (in terms of a conditional
probability) or generate a target sequence given a
source sequence. Along with the new architecture,

EMNLP 2014


score a pair of sequences (in terms of a conditional
probability) or generate a target sequence given a
source sequence. Along with the new architecture,
we proposed a novel hidden unit that includes a reset gate and an update gate that adaptively control
how much each hidden unit remembers or forgets
while reading/generating a sequence.
We evaluated the proposed model with the task
of statistical machine translation, where we used
the RNN Encoder–Decoder to score each phrase
pair in the phrase table. Qualitatively, we were
able to show that the new model is able to capture linguistic regularities in the phrase pairs well
and also that the RNN Encoder–Decoder is able to
propose well-formed target phrases.
The scores by the RNN Encoder–Decoder were
found to improve the overall translation performance in terms of BLEU scores. Also, we
found that the contribution by the RNN Encoder–
Decoder is rather orthogonal to the existing approach of using neural networks in the SMT system, so that we can improve further the performance by using, for instance, the RNN Encoder–
Decoder and the neural net language model together.
Our qualitative analysis of the trained model
shows that it indeed captures the linguistic regularities in multiple levels i.e. at the word level as
well as phrase level. This suggests that there may

EMNLP 2014


Our qualitative analysis of the trained model
shows that it indeed captures the linguistic regularities in multiple levels i.e. at the word level as
well as phrase level. This suggests that there may
be more natural language related applications that
may beneﬁt from the proposed RNN Encoder–
Decoder.
The proposed architecture has large potential
for further improvement and analysis. One approach that was not investigated here is to replace the whole, or a part of the phrase table by
letting the RNN Encoder–Decoder propose target
phrases. Also, noting that the proposed model is
not limited to being used with written language,
it will be an important future research to apply the
proposed architecture to other applications such as
speech transcription.
Acknowledgments
KC, BM, CG, DB and YB would like to thank
NSERC, Calcul Qu ´ebec, Compute Canada, the
Canada Research Chairs and CIFAR. FB and HS
were partially funded by the European Commis-sion under the project MateCat, and by DARPA
under the BOLT project.
References
[Auli et al.2013] Michael Auli, Michel Galley, Chris
Quirk, and Geoffrey Zweig. 2013. Joint language
and translation modeling with recurrent neural networks. In Proceedings of the ACL Conference on
Empirical Methods in Natural Language Processing

EMNLP 2014


Quirk, and Geoffrey Zweig. 2013. Joint language
and translation modeling with recurrent neural networks. In Proceedings of the ACL Conference on
Empirical Methods in Natural Language Processing
(EMNLP) , pages 1044–1054.
[Axelrod et al.2011] Amittai Axelrod, Xiaodong He,
and Jianfeng Gao. 2011. Domain adaptation via
pseudo in-domain data selection. In Proceedings of
the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 355–362.
[Bastien et al.2012] Fr ´ed´eric Bastien, Pascal Lamblin,
Razvan Pascanu, James Bergstra, Ian J. Goodfellow,
Arnaud Bergeron, Nicolas Bouchard, and Yoshua
Bengio. 2012. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.
[Bengio et al.2003] Yoshua Bengio, R ´ejean Ducharme,
Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn.
Res., 3:1137–1155, March.

EMNLP 2014


Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn.
Res., 3:1137–1155, March.
[Bengio et al.2013] Y . Bengio, N. BoulangerLewandowski, and R. Pascanu. 2013. Advances
in optimizing recurrent networks. In Proceedings
of the 38th International Conference on Acoustics,
Speech, and Signal Processing (ICASSP 2013) ,
May.
[Bergstra et al.2010] James Bergstra, Olivier Breuleux,
Fr´ed´eric Bastien, Pascal Lamblin, Razvan Pascanu,
Guillaume Desjardins, Joseph Turian, David WardeFarley, and Yoshua Bengio. 2010. Theano: a CPU
and GPU math expression compiler. In Proceedings
of the Python for Scientiﬁc Computing Conference
(SciPy) , June. Oral Presentation.
[Chandar et al.2014] Sarath Chandar, Stanislas Lauly,
Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. arXiv: 1402.1454 [cs.CL] , February.

EMNLP 2014


[Dahl et al.2012] George E. Dahl, Dong Yu, Li Deng,
and Alex Acero. 2012. Context-dependent pretrained deep neural networks for large vocabulary
speech recognition. IEEE Transactions on Audio,
Speech, and Language Processing , 20(1):33–42.
[Devlin et al.2014] Jacob Devlin, Rabih Zbib,
Zhongqiang Huang, Thomas Lamar, Richard
Schwartz, , and John Makhoul. 2014. Fast and
robust neural network joint models for statistical
machine translation. In Proceedings of the ACL
2014 Conference , ACL ’14, pages 1370–1380.
[Gao et al.2013] Jianfeng Gao, Xiaodong He, Wen tau
Yih, and Li Deng. 2013. Learning semantic representations for the phrase translation model. Technical report, Microsoft Research.
[Glorot et al.2011] X. Glorot, A. Bordes, and Y . Bengio. 2011. Deep sparse rectiﬁer neural networks. In
AISTATS’2011 .
[Goodfellow et al.2013] Ian J. Goodfellow, David
Warde-Farley, Mehdi Mirza, Aaron Courville, and
Yoshua Bengio. 2013. Maxout networks. In
ICML’2013 .

EMNLP 2014


Warde-Farley, Mehdi Mirza, Aaron Courville, and
Yoshua Bengio. 2013. Maxout networks. In
ICML’2013 .
[Graves2012] Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks .
Studies in Computational Intelligence. Springer.
[Hochreiter and Schmidhuber1997] S. Hochreiter and
J. Schmidhuber. 1997. Long short-term memory.
Neural Computation , 9(8):1735–1780.
[Kalchbrenner and Blunsom2013] Nal Kalchbrenner
and Phil Blunsom. 2013. Two recurrent continuous
translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language
Processing (EMNLP) , pages 1700–1709.
[Koehn et al.2003] Philipp Koehn, Franz Josef Och,
and Daniel Marcu. 2003. Statistical phrase-based
translation. In Proceedings of the 2003 Conference
of the North American Chapter of the Association
for Computational Linguistics on Human Language
Technology - Volume 1 , NAACL ’03, pages 48–54.
[Koehn2005] P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Machine

EMNLP 2014


[Koehn2005] P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Machine
Translation Summit X , pages 79–86, Phuket, Thailand.
[Krizhevsky et al.2012] Alex Krizhevsky, Ilya
Sutskever, and Geoffrey Hinton. 2012. ImageNet classiﬁcation with deep convolutional neural
networks. In Advances in Neural Information
Processing Systems 25 (NIPS’2012) .
[Marcu and Wong2002] Daniel Marcu and William
Wong. 2002. A phrase-based, joint probability
model for statistical machine translation. In Proceedings of the ACL-02 Conference on Empirical
Methods in Natural Language Processing - Volume
10, EMNLP ’02, pages 133–139.
[Mikolov et al.2013] Tomas Mikolov, Ilya Sutskever,
Kai Chen, Greg Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and
their compositionality. In Advances in Neural Information Processing Systems 26 , pages 3111–3119.
[Moore and Lewis2010] Robert C. Moore and William
Lewis. 2010. Intelligent selection of language
model training data. In Proceedings of the ACL
2010 Conference Short Papers , ACLShort ’10,

EMNLP 2014


Lewis. 2010. Intelligent selection of language
model training data. In Proceedings of the ACL
2010 Conference Short Papers , ACLShort ’10,
pages 220–224, Stroudsburg, PA, USA.[Pascanu et al.2014] R. Pascanu, C. Gulcehre, K. Cho,
and Y . Bengio. 2014. How to construct deep recurrent neural networks. In Proceedings of the Second
International Conference on Learning Representations (ICLR 2014) , April.
[Saxe et al.2014] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. 2014. Exact solutions
to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of the Second
International Conference on Learning Representations (ICLR 2014) , April.
[Schwenk et al.2006] Holger Schwenk, Marta R. CostaJuss`a, and Jos ´e A. R. Fonollosa. 2006. Continuous
space language models for the iwslt 2006 task. In
IWSLT , pages 166–173.
[Schwenk2007] Holger Schwenk. 2007. Continuous
space language models. Comput. Speech Lang. ,
21(3):492–518, July.
[Schwenk2012] Holger Schwenk. 2012. Continuous

EMNLP 2014


space language models. Comput. Speech Lang. ,
21(3):492–518, July.
[Schwenk2012] Holger Schwenk. 2012. Continuous
space translation models for phrase-based statistical machine translation. In Martin Kay and Christian Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics
(COLIN) , pages 1071–1080.
[Socher et al.2011] Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y . Ng, and Christopher D.
Manning. 2011. Dynamic pooling and unfolding
recursive autoencoders for paraphrase detection. In
Advances in Neural Information Processing Systems
24.
[Son et al.2012] Le Hai Son, Alexandre Allauzen, and
Franc ¸ois Yvon. 2012. Continuous space translation models with neural networks. In Proceedings of
the 2012 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies , NAACL HLT ’12,
pages 39–48, Stroudsburg, PA, USA.
[van der Maaten2013] Laurens van der Maaten. 2013.
Barnes-hut-sne. In Proceedings of the First International Conference on Learning Representations
(ICLR 2013) , May.

EMNLP 2014


Barnes-hut-sne. In Proceedings of the First International Conference on Learning Representations
(ICLR 2013) , May.
[Vaswani et al.2013] Ashish Vaswani, Yinggong Zhao,
Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. Proceedings of the Conference
on Empirical Methods in Natural Language Processing , pages 1387–1392.
[Zeiler2012] Matthew D. Zeiler. 2012. ADADELTA:
an adaptive learning rate method. Technical report,
arXiv 1212.5701.
[Zou et al.2013] Will Y . Zou, Richard Socher,
Daniel M. Cer, and Christopher D. Manning.
2013. Bilingual word embeddings for phrase-based
machine translation. In Proceedings of the ACL
Conference on Empirical Methods in Natural
Language Processing (EMNLP) , pages 1393–1398.
A RNN Encoder–Decoder
In this document, we describe in detail the architecture of the RNN Encoder–Decoder used in the experiments.
Let us denote an source phrase by X= (x1;x2;:::;xN)and a target phrase by Y=

EMNLP 2014


Let us denote an source phrase by X= (x1;x2;:::;xN)and a target phrase by Y=
(y1;y2;:::;yM). Each phrase is a sequence of K-dimensional one-hot vectors, such that only one
element of the vector is 1and all the others are 0. The index of the active ( 1) element indicates the word
represented by the vector.
A.1 Encoder
Each word of the source phrase is embedded in a 500-dimensional vector space: e(xi)2R500.e(x)is
used in Sec. 4.4 to visualize the words.
The hidden state of an encoder consists of 1000 hidden units, and each one of them at time tis
computed by
hhti
j=zjhht1i
j + (1zj)~hhti
j;
where
~hhti
j= tanh
[We(xt)]j+
U
rhht1i
j
;
zj=
[Wze(xt)]j+
Uzhht1i
j
;
rj=
[Wre(xt)]j+
Urhht1i
j
:

EMNLP 2014


j
;
rj=
[Wre(xt)]j+
Urhht1i
j
:
andare a logistic sigmoid function and an element-wise multiplication, respectively. To make the
equations uncluttered, we omit biases. The initial hidden state hh0i
jis ﬁxed to 0.
Once the hidden state at the Nstep (the end of the source phrase) is computed, the representation of
the source phrase cis
c= tanh
VhhNi
:
A.1.1 Decoder
The decoder starts by initializing the hidden state with
h0h0i= tanh
V0c
;
where we will use0to distinguish parameters of the decoder from those of the encoder.
The hidden state at time tof the decoder is computed by
h0hti
j=z0jh0ht1i
j + (1z0j)~h0hti
j;
where
~h0hti
j= tanh
W0e(yt1)
j+r0j
U0h0
ht1i+Cc
;
z0j=

EMNLP 2014


j+r0j
U0h0
ht1i+Cc
;
z0j=
W0ze(yt1)
j+
U0zh0
ht1i
j+ [Czc]j
;
r0j=
W0re(yt1)
j+
U0rh0
ht1i
j+ [Crc]j
;
ande(y0)is an all-zero vector. Similarly to the case of the encoder, e(y)is an embedding of a target
word.
Unlike the encoder which simply encodes the source phrase, the decoder is learned to generate a target
phrase. At each time t, the decoder computes the probability of generating j-th word by
p(yt;j= 1jyt1;:::;y1;X) =exp
gjshti
PK
j0=1exp
gj0shti;
where thei-element of shtiis
shti
i= maxn
s0hti
2i1;s0hti
2io
and
s0hti=Ohh0hti+Oyyt1+Occ:

EMNLP 2014


s0hti
2i1;s0hti
2io
and
s0hti=Ohh0hti+Oyyt1+Occ:
In short, the shti
iis a so-called maxout unit.
For the computational efﬁciency, instead of a single-matrix output weight G, we use a product of two
matrices such that
G=GlGr;
where Gl2RK500andGr2R5001000.
B Word and Phrase Representations
Here, we show enlarged plots of the word and phrase representations in Figs. 4–5.
Figure 6: 2–D embedding of the learned word representation. The top left one shows the full embedding space, while the other three ﬁgures show the zoomed-in
view of speciﬁc regions (color–coded).
Figure 7: 2–D embedding of the learned phrase representation. The top left one shows the full representation space (1000 randomly selected points), while the
other three ﬁgures show the zoomed-in view of speciﬁc regions (color–coded).

EMNLP 2014


1
Spatial Pyramid Pooling in Deep Convolutional
Networks for Visual Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun
Abstract —Existing deep convolutional neural networks (CNNs) require a ﬁxed-size ( e.g., 224224) input image. This requirement is “artiﬁcial” and may reduce the recognition accuracy for the images or sub-images of an arbitrary size/scale. In this
work, we equip the networks with another pooling strategy, “spatial pyramid pooling”, to eliminate the above requirement. The
new network structure, called SPP-net, can generate a ﬁxed-length representation regardless of image size/scale. Pyramid
pooling is also robust to object deformations. With these advantages, SPP-net should in general improve all CNN-based image
classiﬁcation methods. On the ImageNet 2012 dataset, we demonstrate that SPP-net boosts the accuracy of a variety of CNN
architectures despite their different designs. On the Pascal VOC 2007 and Caltech101 datasets, SPP-net achieves state-of-theart classiﬁcation results using a single full-image representation and no ﬁne-tuning.

Summary:
Existing deep convolutional neural networks (CNNs) require a fixed-size
(e.g., 224x224) input image. This requirement is "artificial" and may reduce
the recognition accuracy for the images or sub-images of an arbitrary
size/scale. In this work, we equip the networks with another pooling strategy,
"spatial pyramid pooling", to eliminate the above requirement. The new network
structure, called SPP-net, can generate a fixed-length representation
regardless of image size/scale. Pyramid pooling is also robust to object
deformations. With these advantages, SPP-net should in general improve all
CNN-based image classification methods. On the ImageNet 2012 dataset, we
demonstrate that SPP-net boosts the accuracy of a variety of CNN architectures
despite their different designs. On the Pascal VOC 2007 and Caltech101
datasets, SPP-net achieves state-of-the-art classification results using a
single full-image representation and no fine-tuning.
  The power of SPP-net is also significant in object detection. Using SPP-net,
we compute the feature maps from the entire image only once, and then pool
features in arbitrary regions (sub-images) to generate fixed-length
representations for training the detectors. This method avoids repeatedly
computing the convolutional features. In processing test images, our method is
24-102x faster than the R-CNN method, while achieving better or comparable
accuracy on Pascal VOC 2007.
  In ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2014, our
methods rank #2 in object detection and #3 in image classification among all 38
teams. This manuscript also introduces the improvement made for this
competition.

Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

This manuscript is the accepted version for IEEE Transactions on
  Pattern Analysis and Machine Intelligence (TPAMI) 2015. See Changelog


The power of SPP-net is also signiﬁcant in object detection. Using SPP-net, we compute the feature maps from the entire
image only once, and then pool features in arbitrary regions (sub-images) to generate ﬁxed-length representations for training
the detectors. This method avoids repeatedly computing the convolutional features. In processing test images, our method is
24-102faster than the R-CNN method, while achieving better or comparable accuracy on Pascal VOC 2007.
In ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2014, our methods rank #2 in object detection and #3 in
image classiﬁcation among all 38 teams. This manuscript also introduces the improvement made for this competition.
Index Terms —Convolutional Neural Networks, Spatial Pyramid Pooling, Image Classiﬁcation, Object Detection
F
1 I NTRODUCTION
We are witnessing a rapid, revolutionary change in
our vision community, mainly caused by deep convolutional neural networks (CNNs) [1] and the availability of large scale training data [2]. Deep-networksbased approaches have recently been substantially
improving upon the state of the art in image classiﬁcation [3], [4], [5], [6], object detection [7], [8], [5],


improving upon the state of the art in image classiﬁcation [3], [4], [5], [6], object detection [7], [8], [5],
many other recognition tasks [9], [10], [11], [12], and
even non-recognition tasks.
However, there is a technical issue in the training
and testing of the CNNs: the prevalent CNNs require
aﬁxed input image size ( e.g., 224224), which limits
both the aspect ratio and the scale of the input image.
When applied to images of arbitrary sizes, current
methods mostly ﬁt the input image to the ﬁxed size,
either via cropping [3], [4] or via warping [13], [7],
as shown in Figure 1 (top). But the cropped region
may not contain the entire object, while the warped
content may result in unwanted geometric distortion.
Recognition accuracy can be compromised due to the
content loss or distortion. Besides, a pre-deﬁned scale
K. He and J. Sun are with Microsoft Research, Beijing, China. E-mail:
fkahe,jiansung@microsoft.com
X. Zhang is with Xi’an Jiaotong University, Xi’an, China. Email:


fkahe,jiansung@microsoft.com
X. Zhang is with Xi’an Jiaotong University, Xi’an, China. Email:
xyz.clx@stu.xjtu.edu.cn
S. Ren is with University of Science and Technology of China, Hefei,
China. Email: sqren@mail.ustc.edu.cn
This work was done when X. Zhang and S. Ren were interns at Microsoft
Research.
crop warp
spatial pyramid poolingcrop / warp
conv layers image fc layers outputimage conv layers fc layers outputFigure 1: Top: cropping or warping to ﬁt a ﬁxed
size. Middle: a conventional CNN. Bottom: our spatial
pyramid pooling network structure.
may not be suitable when object scales vary. Fixing
input sizes overlooks the issues involving scales.
So why do CNNs require a ﬁxed input size? A CNN
mainly consists of two parts: convolutional layers,
and fully-connected layers that follow. The convolutional layers operate in a sliding-window manner
and output feature maps which represent the spatial
arrangement of the activations (Figure 2). In fact, convolutional layers do not require a ﬁxed image size and


and output feature maps which represent the spatial
arrangement of the activations (Figure 2). In fact, convolutional layers do not require a ﬁxed image size and
can generate feature maps of any sizes. On the other
hand, the fully-connected layers need to have ﬁxedsize/length input by their deﬁnition. Hence, the ﬁxedsize constraint comes only from the fully-connected
layers, which exist at a deeper stage of the network.
In this paper, we introduce a spatial pyramid pooling (SPP) [14], [15] layer to remove the ﬁxed-size
constraint of the network. Speciﬁcally, we add anarXiv:1406.4729v4  [cs.CV]  23 Apr 2015
2
SPP layer on top of the last convolutional layer. The
SPP layer pools the features and generates ﬁxedlength outputs, which are then fed into the fullyconnected layers (or other classiﬁers). In other words,
we perform some information “aggregation” at a
deeper stage of the network hierarchy (between convolutional layers and fully-connected layers) to avoid
the need for cropping or warping at the beginning.


deeper stage of the network hierarchy (between convolutional layers and fully-connected layers) to avoid
the need for cropping or warping at the beginning.
Figure 1 (bottom) shows the change of the network
architecture by introducing the SPP layer. We call the
new network structure SPP-net .
Spatial pyramid pooling [14], [15] (popularly
known as spatial pyramid matching or SPM [15]), as
an extension of the Bag-of-Words (BoW) model [16],
is one of the most successful methods in computer
vision. It partitions the image into divisions from
ﬁner to coarser levels, and aggregates local features
in them. SPP has long been a key component in the
leading and competition-winning systems for classiﬁcation ( e.g., [17], [18], [19]) and detection ( e.g., [20])
before the recent prevalence of CNNs. Nevertheless,
SPP has not been considered in the context of CNNs.
We note that SPP has several remarkable properties
for deep CNNs: 1) SPP is able to generate a ﬁxedlength output regardless of the input size, while the
sliding window pooling used in the previous deep
networks [3] cannot; 2) SPP uses multi-level spatial


sliding window pooling used in the previous deep
networks [3] cannot; 2) SPP uses multi-level spatial
bins, while the sliding window pooling uses only
a single window size. Multi-level pooling has been
shown to be robust to object deformations [15]; 3) SPP
can pool features extracted at variable scales thanks
to the ﬂexibility of input scales. Through experiments
we show that all these factors elevate the recognition
accuracy of deep networks.
SPP-net not only makes it possible to generate representations from arbitrarily sized images/windows
for testing, but also allows us to feed images with
varying sizes or scales during training. Training with
variable-size images increases scale-invariance and
reduces over-ﬁtting. We develop a simple multi-size
training method. For a single network to accept
variable input sizes, we approximate it by multiple
networks that share all parameters, while each of
these networks is trained using a ﬁxed input size. In
each epoch we train the network with a given input
size, and switch to another input size for the next
epoch. Experiments show that this multi-size training
converges just as the traditional single-size training,
and leads to better testing accuracy.


epoch. Experiments show that this multi-size training
converges just as the traditional single-size training,
and leads to better testing accuracy.
The advantages of SPP are orthogonal to the speciﬁc
CNN designs. In a series of controlled experiments on
the ImageNet 2012 dataset, we demonstrate that SPP
improves four different CNN architectures in existing
publications [3], [4], [5] (or their modiﬁcations), over
the no-SPP counterparts. These architectures have
various ﬁlter numbers/sizes, strides, depths, or other
designs. It is thus reasonable for us to conjecture
that SPP should improve more sophisticated (deeperand larger) convolutional architectures. SPP-net also
shows state-of-the-art classiﬁcation results on Caltech101 [21] and Pascal VOC 2007 [22] using only a
single full-image representation and no ﬁne-tuning.
SPP-net also shows great strength in object detection. In the leading object detection method R-CNN
[7], the features from candidate windows are extracted
via deep convolutional networks. This method shows
remarkable detection accuracy on both the VOC and
ImageNet datasets. But the feature computation in RCNN is time-consuming, because it repeatedly applies


remarkable detection accuracy on both the VOC and
ImageNet datasets. But the feature computation in RCNN is time-consuming, because it repeatedly applies
the deep convolutional networks to the raw pixels
of thousands of warped regions per image. In this
paper, we show that we can run the convolutional
layers only once on the entire image (regardless of
the number of windows), and then extract features
by SPP-net on the feature maps. This method yields
a speedup of over one hundred times over R-CNN.
Note that training/running a detector on the feature
maps (rather than image regions) is actually a more
popular idea [23], [24], [20], [5]. But SPP-net inherits
the power of the deep CNN feature maps and also the
ﬂexibility of SPP on arbitrary window sizes, which
leads to outstanding accuracy and efﬁciency. In our
experiment, the SPP-net-based system (built upon the
R-CNN pipeline) computes features 24-102 faster
than R-CNN, while has better or comparable accuracy.
With the recent fast proposal method of EdgeBoxes
[25], our system takes 0.5 seconds processing an image
(including all steps ). This makes our method practical
for real-world applications.


[25], our system takes 0.5 seconds processing an image
(including all steps ). This makes our method practical
for real-world applications.
A preliminary version of this manuscript has been
published in ECCV 2014. Based on this work, we
attended the competition of ILSVRC 2014 [26], and
ranked #2 in object detection and #3 in image classiﬁcation (both are provided-data-only tracks) among
all 38 teams. There are a few modiﬁcations made
for ILSVRC 2014. We show that the SPP-nets can
boost various networks that are deeper and larger
(Sec. 3.1.2-3.1.4) over the no-SPP counterparts. Further, driven by our detection framework, we ﬁnd
that multi-view testing on feature maps with ﬂexibly
located/sized windows (Sec. 3.1.5) can increase the
classiﬁcation accuracy. This manuscript also provides
the details of these modiﬁcations.
We have released the code to facilitate future research ( http://research.microsoft.com/en-us/um/people/kahe/ ).
2 D EEP NETWORKS WITH SPATIAL PYRAMIDPOOLING
2.1 Convolutional Layers and Feature Maps


2 D EEP NETWORKS WITH SPATIAL PYRAMIDPOOLING
2.1 Convolutional Layers and Feature Maps
Consider the popular seven-layer architectures [3], [4].
The ﬁrst ﬁve layers are convolutional, some of which
are followed by pooling layers. These pooling layers
can also be considered as “convolutional”, in the sense
that they are using sliding windows. The last two
3

filter #175
filter #55
(a) image (b) feature maps (c) strongest activations

filter #66
filter #118
(a) image (b) feature maps (c) strongest activations
Figure 2: Visualization of the feature maps. (a) Two images in Pascal VOC 2007. (b) The feature maps of some
conv 5ﬁlters. The arrows indicate the strongest responses and their corresponding positions in the images.
(c) The ImageNet images that have the strongest responses of the corresponding ﬁlters. The green rectangles
mark the receptive ﬁelds of the strongest responses.
layers are fully connected, with an N-way softmax as
the output, where N is the number of categories.
The deep network described above needs a ﬁxed


layers are fully connected, with an N-way softmax as
the output, where N is the number of categories.
The deep network described above needs a ﬁxed
image size. However, we notice that the requirement
of ﬁxed sizes is only due to the fully-connected layers
that demand ﬁxed-length vectors as inputs. On the
other hand, the convolutional layers accept inputs of
arbitrary sizes. The convolutional layers use sliding
ﬁlters, and their outputs have roughly the same aspect
ratio as the inputs. These outputs are known as feature
maps [1] - they involve not only the strength of the
responses, but also their spatial positions.
In Figure 2, we visualize some feature maps. They
are generated by some ﬁlters of the conv 5layer. Figure 2(c) shows the strongest activated images of these
ﬁlters in the ImageNet dataset. We see a ﬁlter can be
activated by some semantic content. For example, the
55-th ﬁlter (Figure 2, bottom left) is most activated by
a circle shape; the 66-th ﬁlter (Figure 2, top right) is


a circle shape; the 66-th ﬁlter (Figure 2, top right) is
most activated by a ^-shape; and the 118-th ﬁlter (Figure 2, bottom right) is most activated by a _-shape.
These shapes in the input images (Figure 2(a)) activate
the feature maps at the corresponding positions (the
arrows in Figure 2).
It is worth noticing that we generate the feature
maps in Figure 2 without ﬁxing the input size. These
feature maps generated by deep convolutional layers are analogous to the feature maps in traditional
methods [27], [28]. In those methods, SIFT vectors
[29] or image patches [28] are densely extracted and
then encoded, e.g., by vector quantization [16], [15],
[30], sparse coding [17], [18], or Fisher kernels [19].
These encoded features consist of the feature maps,
and are then pooled by Bag-of-Words (BoW) [16] or
spatial pyramids [14], [15]. Analogously, the deep
convolutional features can be pooled in a similar way.
2.2 The Spatial Pyramid Pooling Layer
The convolutional layers accept arbitrary input sizes,


convolutional features can be pooled in a similar way.
2.2 The Spatial Pyramid Pooling Layer
The convolutional layers accept arbitrary input sizes,
but they produce outputs of variable sizes. The classiﬁers (SVM/softmax) or fully-connected layers require
convolutional layersfeature maps of conv 5
(arbitrary size)
fixed-length representation
input image16×256-d 4×256-d 256-d…...
…...
spatial pyramid pooling layerfully-connected layers (fc 6, fc 7)Figure 3: A network structure with a spatial pyramid
pooling layer . Here 256 is the ﬁlter number of the
conv 5layer, and conv 5is the last convolutional layer.
ﬁxed-length vectors. Such vectors can be generated
by the Bag-of-Words (BoW) approach [16] that pools
the features together. Spatial pyramid pooling [14],
[15] improves BoW in that it can maintain spatial
information by pooling in local spatial bins. These
spatial bins have sizes proportional to the image size,
so the number of bins is ﬁxed regardless of the image
size. This is in contrast to the sliding window pooling
of the previous deep networks [3], where the number


so the number of bins is ﬁxed regardless of the image
size. This is in contrast to the sliding window pooling
of the previous deep networks [3], where the number
of sliding windows depends on the input size.
To adopt the deep network for images of arbitrary sizes, we replace the last pooling layer ( e.g.,
pool 5, after the last convolutional layer) with a spatial
pyramid pooling layer . Figure 3 illustrates our method.
In each spatial bin, we pool the responses of each
ﬁlter (throughout this paper we use max pooling).
The outputs of the spatial pyramid pooling are kMdimensional vectors with the number of bins denoted
asM(kis the number of ﬁlters in the last convolutional layer). The ﬁxed-dimensional vectors are the
input to the fully-connected layer.
With spatial pyramid pooling, the input image can
4
be of any sizes. This not only allows arbitrary aspect
ratios, but also allows arbitrary scales. We can resize
the input image to any scale ( e.g.,min(w;h)=180, 224,
...) and apply the same deep network. When the
input image is at different scales, the network (with
the same ﬁlter sizes) will extract features at different