...) and apply the same deep network. When the
input image is at different scales, the network (with
the same ﬁlter sizes) will extract features at different
scales. The scales play important roles in traditional
methods, e.g., the SIFT vectors are often extracted at
multiple scales [29], [27] (determined by the sizes of
the patches and Gaussian ﬁlters). We will show that
the scales are also important for the accuracy of deep
networks.
Interestingly, the coarsest pyramid level has a single
bin that covers the entire image. This is in fact a
“global pooling” operation, which is also investigated
in several concurrent works. In [31], [32] a global
average pooling is used to reduce the model size
and also reduce overﬁtting; in [33], a global average
pooling is used on the testing stage after all fc layers
to improve accuracy; in [34], a global max pooling is
used for weakly supervised object recognition. The
global pooling operation corresponds to the traditional Bag-of-Words method.
2.3 Training the Network
Theoretically, the above network structure can be
trained with standard back-propagation [1], regardless of the input image size. But in practice the GPU


Theoretically, the above network structure can be
trained with standard back-propagation [1], regardless of the input image size. But in practice the GPU
implementations (such as cuda-convnet [3] and Caffe
[35]) are preferably run on ﬁxed input images. Next
we describe our training solution that takes advantage
of these GPU implementations while still preserving
the spatial pyramid pooling behaviors.
Single-size training
As in previous works, we ﬁrst consider a network taking a ﬁxed-size input (224 224) cropped from images.
The cropping is for the purpose of data augmentation.
For an image with a given size, we can pre-compute
the bin sizes needed for spatial pyramid pooling.
Consider the feature maps after conv 5that have a size
ofaa(e.g., 1313). With a pyramid level of nn
bins, we implement this pooling level as a sliding
window pooling, where the window size win=da=ne
and stride str=ba=ncwithdeandbcdenoting
ceiling and ﬂoor operations. With an l-level pyramid,
we implement lsuch layers. The next fully-connected
layer (fc 6) will concatenate the loutputs. Figure 4

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

Summary:
Existing deep convolutional neural networks (CNNs) require a fixed-size
(e.g., 224x224) input image. This requirement is "artificial" and may reduce
the recognition accuracy for the images or sub-images of an arbitrary
size/scale. In this work, we equip the networks with another pooling strategy,
"spatial pyramid pooling", to eliminate the above requirement. The new network
structure, called SPP-net, can generate a fixed-length representation
regardless of image size/scale. Pyramid pooling is also robust to object
deformations. With these advantages, SPP-net should in general improve all
CNN-based image classification methods. On the ImageNet 2012 dataset, we
demonstrate that SPP-net boosts the accuracy of a variety of CNN architectures
despite their different designs. On the Pascal VOC 2007 and Caltech101
datasets, SPP-net achieves state-of-the-art classification results using a
single full-image representation and no fine-tuning.
  The power of SPP-net is also significant in object detection. Using SPP-net,
we compute the feature maps from the entire image only once, and then pool
features in arbitrary regions (sub-images) to generate fixed-length
representations for training the detectors. This method avoids repeatedly
computing the convolutional features. In processing test images, our method is
24-102x faster than the R-CNN method, while achieving better or comparable
accuracy on Pascal VOC 2007.
  In ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2014, our
methods rank #2 in object detection and #3 in image classification among all 38
teams. This manuscript also introduces the improvement made for this
competition.

Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

This manuscript is the accepted version for IEEE Transactions on
  Pattern Analysis and Machine Intelligence (TPAMI) 2015. See Changelog


we implement lsuch layers. The next fully-connected
layer (fc 6) will concatenate the loutputs. Figure 4
shows an example conﬁguration of 3-level pyramid
pooling (33, 22, 11) in the cuda-convnet style [3].
The main purpose of our single-size training is to
enable the multi-level pooling behavior. Experiments
show that this is one reason for the gain of accuracy.
Multi-size training
Our network with SPP is expected to be applied on
images of any sizes. To address the issue of varying
[fc6]
type=fcoutputs=4096inputs=pool3x3,pool2x2,pool1x1[pool1x1]
type=poolpool=maxinputs=conv5sizeX=13stride=13[pool3x3]
type=poolpool=maxinputs=conv5sizeX=5stride=4[pool2x2]
type=poolpool=maxinputs=conv5sizeX=7stride=6Figure 4: An example 3-level pyramid pooling in the
cuda-convnet style [3]. Here sizeX is the size of the
pooling window. This conﬁguration is for a network


cuda-convnet style [3]. Here sizeX is the size of the
pooling window. This conﬁguration is for a network
whose feature map size of conv 5is 1313, so the
pool 33, pool 22, and pool 11layers will have 3 3,
22, and 11 bins respectively.
image sizes in training, we consider a set of predeﬁned sizes. We consider two sizes: 180 180 in addition to 224224. Rather than crop a smaller 180 180
region, we resize the aforementioned 224 224 region
to 180180. So the regions at both scales differ only
in resolution but not in content/layout. For the network to accept 180 180 inputs, we implement another
ﬁxed-size-input (180 180) network. The feature map
size after conv 5isaa= 1010 in this case. Then we
still usewin=da=neandstr=ba=ncto implement
each pyramid pooling level. The output of the spatial
pyramid pooling layer of this 180-network has the
same ﬁxed length as the 224-network. As such, this
180-network has exactly the same parameters as the


same ﬁxed length as the 224-network. As such, this
180-network has exactly the same parameters as the
224-network in each layer. In other words, during
training we implement the varying-input-size SPP-net
by two ﬁxed-size networks that share parameters.
To reduce the overhead to switch from one network
(e.g., 224) to the other ( e.g., 180), we train each full
epoch on one network, and then switch to the other
one (keeping all weights) for the next full epoch. This
is iterated. In experiments, we ﬁnd the convergence
rate of this multi-size training to be similar to the
above single-size training.
The main purpose of our multi-size training is to
simulate the varying input sizes while still leveraging
the existing well-optimized ﬁxed-size implementations. Besides the above two-scale implementation, we
have also tested a variant using ssas input where
sis randomly and uniformly sampled from [180;224]
at each epoch. We report the results of both variants
in the experiment section.
Note that the above single/multi-size solutions are
for training only. At the testing stage, it is straightforward to apply SPP-net on images of any sizes.
5


Note that the above single/multi-size solutions are
for training only. At the testing stage, it is straightforward to apply SPP-net on images of any sizes.
5
model conv 1 conv 2 conv 3 conv 4 conv 5 conv 6 conv 7
ZF-5 9672, str 2 25652, str 2 384323843225632
LRN, pool 32, str 2 LRN, pool 32, str 2 - map size 5555 2727 1313 1313 1313
Convnet*-5 96112, str 4 25652384323843225632
LRN, LRN, pool 32, str 2 pool 32, 2 - map size 5555 2727 1313 1313 1313
Overfeat-5/7 9672, str 2 256525123251232512325123251232
pool 32, str 3, LRN pool 22, str 2
map size 3636 1818 1818 1818 1818 1818 1818
Table 1: Network architectures: ﬁlter number ﬁlter size ( e.g.,9672), ﬁlter stride ( e.g., str 2), pooling window


size ( e.g., pool 32), and the output feature map size ( e.g., map size 5555). LRN represents Local Response
Normalization. The padding is adjusted to produce the expected output feature map size.
3 SPP- NET FOR IMAGE CLASSIFICATION
3.1 Experiments on ImageNet 2012 Classiﬁcation
We train the networks on the 1000-category training
set of ImageNet 2012. Our training algorithm follows
the practices of previous work [3], [4], [36]. The images are resized so that the smaller dimension is 256,
and a 224224 crop is picked from the center or the
four corners from the entire image1. The data are augmented by horizontal ﬂipping and color altering [3].
Dropout [3] is used on the two fully-connected layers.
The learning rate starts from 0.01, and is divided by 10
(twice) when the error plateaus. Our implementation
is based on the publicly available code of cuda-convnet
[3] and Caffe [35]. All networks in this paper can be
trained on a single GeForce GTX Titan GPU (6 GB
memory) within two to four weeks.
3.1.1 Baseline Network Architectures
The advantages of SPP are independent of the convolutional network architectures used. We investigate


memory) within two to four weeks.
3.1.1 Baseline Network Architectures
The advantages of SPP are independent of the convolutional network architectures used. We investigate
four different network architectures in existing publications [3], [4], [5] (or their modiﬁcations), and we
show SPP improves the accuracy of all these architectures. These baseline architectures are in Table 1 and
brieﬂy introduced below:
ZF-5 : this architecture is based on Zeiler and Fergus’s (ZF) “fast” (smaller) model [4]. The number
indicates ﬁve convolutional layers.
Convnet*-5 : this is a modiﬁcation on Krizhevsky
et al.’s network [3]. We put the two pooling layers
after conv 2and conv 3(instead of after conv 1and
conv 2). As a result, the feature maps after each
layer have the same size as ZF-5.
Overfeat-5/7 : this architecture is based on the
Overfeat paper [5], with some modiﬁcations as in
[6]. In contrast to ZF-5/Convnet*-5, this architecture produces a larger feature map ( 1818instead


[6]. In contrast to ZF-5/Convnet*-5, this architecture produces a larger feature map ( 1818instead
of1313) before the last pooling layer. A larger
ﬁlter number (512) is used in conv 3and the following convolutional layers. We also investigate
1. In [3], the four corners are picked from the corners of the
central 256256 crop.a deeper architecture with 7 convolutional layers,
where conv 3to conv 7have the same structures.
In the baseline models, the pooling layer after the last
convolutional layer generates 66feature maps, with
two 4096-d fc layers and a 1000-way softmax layer
following. Our replications of these baseline networks
are in Table 2 (a). We train 70 epochs for ZF-5 and
90 epochs for the others. Our replication of ZF-5 is
better than the one reported in [4]. This gain is because
the corner crops are from the entire image, as is also
reported in [36].
3.1.2 Multi-level Pooling Improves Accuracy
In Table 2 (b) we show the results using singlesize training. The training and testing sizes are both
224224. In these networks, the convolutional layers


In Table 2 (b) we show the results using singlesize training. The training and testing sizes are both
224224. In these networks, the convolutional layers
have the same structures as the corresponding baseline models, whereas the pooling layer after the ﬁnal
convolutional layer is replaced with the SPP layer. For
the results in Table 2, we use a 4-level pyramid. The
pyramid isf66, 33, 22, 11g(totally 50 bins).
For fair comparison, we still use the standard 10view prediction with each view a 224 224 crop. Our
results in Table 2 (b) show considerable improvement
over the no-SPP baselines in Table 2 (a). Interestingly,
the largest gain of top-1 error (1.65%) is given by the
most accurate architecture. Since we are still using the
same 10 cropped views as in (a), these gains are solely
because of multi-level pooling.
It is worth noticing that the gain of multi-level
pooling is notsimply due to more parameters; rather,
it is because the multi-level pooling is robust to the
variance in object deformations and spatial layout
[15]. To show this, we train another ZF-5 network with


it is because the multi-level pooling is robust to the
variance in object deformations and spatial layout
[15]. To show this, we train another ZF-5 network with
a different 4-level pyramid: f44, 33, 22, 11g
(totally 30 bins). This network has fewer parameters
than its no-SPP counterpart, because its fc 6layer has
30256-d inputs instead of 36 256-d. The top-1/top5 errors of this network are 35.06/14.04. This result
is similar to the 50-bin pyramid above (34.98/14.14),
but considerably better than the no-SPP counterpart
(35.99/14.76).
6
top-1 error (%)
ZF-5 Convnet*-5 Overfeat-5 Overfeat-7
(a) no SPP 35.99 34.93 34.13 32.01
(b) SPP single-size trained 34.98 (1.01) 34.38 (0.55) 32.87 (1.26) 30.36 (1.65)
(c) SPP multi-size trained 34.60 (1.39) 33.94 (0.99) 32.26 (1.87) 29.68 (2.33)


top-5 error (%)
ZF-5 Convnet*-5 Overfeat-5 Overfeat-7
(a) no SPP 14.76 13.92 13.52 11.97
(b) SPP single-size trained 14.14 (0.62) 13.54 (0.38) 12.80 (0.72) 11.12 (0.85)
(c) SPP multi-size trained 13.64 (1.12) 13.33 (0.59) 12.33 (1.19) 10.95 (1.02)
Table 2: Error rates in the validation set of ImageNet 2012. All the results are obtained using standard 10-view
testing. In the brackets are the gains over the “no SPP” baselines.
SPP on test view top-1 val
ZF-5, single-size trained 1 crop 38.01
ZF-5, single-size trained 1 full 37.55
ZF-5, multi-size trained 1 crop 37.57
ZF-5, multi-size trained 1 full 37.07
Overfeat-7, single-size trained 1 crop 33.18
Overfeat-7, single-size trained 1 full 32.72
Overfeat-7, multi-size trained 1 crop 32.57


Overfeat-7, single-size trained 1 full 32.72
Overfeat-7, multi-size trained 1 crop 32.57
Overfeat-7, multi-size trained 1 full 31.25
Table 3: Error rates in the validation set of ImageNet
2012 using a single view. The images are resized so
min(w;h) = 256 . The crop view is the central 224 224
of the image.
3.1.3 Multi-size Training Improves Accuracy
Table 2 (c) shows our results using multi-size training.
The training sizes are 224 and 180, while the testing
size is still 224. We still use the standard 10-view
prediction. The top-1/top-5 errors of all architectures
further drop. The top-1 error of SPP-net (Overfeat-7)
drops to 29.68%, which is 2.33% better than its noSPP counterpart and 0.68% better than its single-size
trained counterpart.
Besides using the two discrete sizes of 180 and
224, we have also evaluated using a random size
uniformly sampled from [180;224]. The top-1/5 error
of SPP-net (Overfeat-7) is 30.06%/10.96%. The top1 error is slightly worse than the two-size version,


of SPP-net (Overfeat-7) is 30.06%/10.96%. The top1 error is slightly worse than the two-size version,
possibly because the size of 224 (which is used for
testing) is visited less. But the results are still better
the single-size version.
There are previous CNN solutions [5], [36] that deal
with various scales/sizes, but they are mostly based
on testing. In Overfeat [5] and Howard’s method [36],
the single network is applied at multiple scales in the
testing stage, and the scores are averaged. Howard
further trains two different networks on low/highresolution image regions and averages the scores. To
our knowledge, our method is the ﬁrst one that trains
a single network with input images of multiple sizes.3.1.4 Full-image Representations Improve Accuracy
Next we investigate the accuracy of the full-image
views. We resize the image so that min(w;h)=256
while maintaining its aspect ratio. The SPP-net is
applied on this full image to compute the scores of
the full view. For fair comparison, we also evaluate
the accuracy of the single view in the center 224 224
crop (which is used in the above evaluations). The
comparisons of single-view testing accuracy are in


the accuracy of the single view in the center 224 224
crop (which is used in the above evaluations). The
comparisons of single-view testing accuracy are in
Table 3. Here we evaluate ZF-5/Overfeat-7. The top-1
error rates are all reduced by the full-view representation. This shows the importance of maintaining the
complete content. Even though our network is trained
using square images only, it generalizes well to other
aspect ratios.
Comparing Table 2 and Table 3, we ﬁnd that the
combination of multiple views is substantially better
than the single full-image view. However, the fullimage representations are still of good merits. First,
we empirically ﬁnd that (discussed in the next subsection) even for the combination of dozens of views,
the additional two full-image views (with ﬂipping)
can still boost the accuracy by about 0.2%. Second,
the full-image view is methodologically consistent
with the traditional methods [15], [17], [19] where the
encoded SIFT vectors of the entire image are pooled
together. Third, in other applications such as image
retrieval [37], an image representation, rather than a
classiﬁcation score, is required for similarity ranking.
A full-image representation can be preferred.


retrieval [37], an image representation, rather than a
classiﬁcation score, is required for similarity ranking.
A full-image representation can be preferred.
3.1.5 Multi-view Testing on Feature Maps
Inspired by our detection algorithm (described in
the next section), we further propose a multi-view
testing method on the feature maps. Thanks to the
ﬂexibility of SPP , we can easily extract the features
from windows (views) of arbitrary sizes from the
convolutional feature maps.
On the testing stage, we resize an image so
min(w;h) =swheresrepresents a predeﬁned scale
(like 256). Then we compute the convolutional feature maps from the entire image. For the usage of
7
method test scales test views top-1 val top-5 val top-5 test
Krizhevsky et al. [3] 1 10 40.7 18.2
Overfeat (fast) [5] 1 - 39.01 16.97
Overfeat (fast) [5] 6 - 38.12 16.27
Overfeat (big) [5] 4 - 35.74 14.18
Howard (base) [36] 3 162 37.0 15.8
Howard (high-res) [36] 3 162 36.8 16.2


Howard (base) [36] 3 162 37.0 15.8
Howard (high-res) [36] 3 162 36.8 16.2
Zeiler & Fergus (ZF) (fast) [4] 1 10 38.4 16.5
Zeiler & Fergus (ZF) (big) [4] 1 10 37.5 16.0
Chatﬁeld et al. [6] 1 10 - 13.1
ours (SPP O-7) 1 10 29.68 10.95
ours (SPP O-7) 6 96+2full 27.86 9.14 9.08
Table 4: Error rates in ImageNet 2012. All the results are based on a single network . The number of views in
Overfeat depends on the scales and strides, for which there are several hundreds at the ﬁnest scale.
ﬂipped views, we also compute the feature maps of
the ﬂipped image. Given any view (window) in the
image, we map this window to the feature maps (the
way of mapping is in Appendix), and then use SPP
to pool the features from this window (see Figure 5).
The pooled features are then fed into the fc layers
to compute the softmax score of this window. These
scores are averaged for the ﬁnal prediction. For the


The pooled features are then fed into the fc layers
to compute the softmax score of this window. These
scores are averaged for the ﬁnal prediction. For the
standard 10-view, we use s= 256 and the views
are 224224 windows on the corners or center. Experiments show that the top-5 error of the 10-view
prediction on feature maps is within 0.1% around the
original 10-view prediction on image crops.
We further apply this method to extract multiple
views from multiple scales. We resize the image to six
scaless2 f224;256;300;360;448;560gand compute
the feature maps on the entire image for each scale.
We use 224224 as the view size for any scale,
so these views have different relative sizes on the
original image for different scales. We use 18 views
for each scale: one at the center, four at the corners,
and four on the middle of each side, with/without
ﬂipping (when s= 224 there are 6 different views).
The combination of these 96 views reduces the top-5
error from 10.95% to 9.36%. Combining the two fullimage views (with ﬂipping) further reduces the top-5
error to 9.14%.


error from 10.95% to 9.36%. Combining the two fullimage views (with ﬂipping) further reduces the top-5
error to 9.14%.
In the Overfeat paper [5], the views are also extracted from the convolutional feature maps instead
of image crops. However, their views cannot have arbitrary sizes; rather, the windows are those where the
pooled features match the desired dimensionality. We
empirically ﬁnd that these restricted windows are less
beneﬁcial than our ﬂexibly located/sized windows.
3.1.6 Summary and Results for ILSVRC 2014
In Table 4 we compare with previous state-of-theart methods. Krizhevsky et al .’s [3] is the winning
method in ILSVRC 2012; Overfeat [5], Howard’s [36],
and Zeiler and Fergus’s [4] are the leading methodsrank team top-5 test
1 GoogLeNet [32] 6.66
2 VGG [33] 7.32
3 ours 8.06
4 Howard 8.11
5 DeeperVision 9.50
6 NUS-BST 9.79
7 TTIC ECP 10.22


3 ours 8.06
4 Howard 8.11
5 DeeperVision 9.50
6 NUS-BST 9.79
7 TTIC ECP 10.22
Table 5: The competition results of ILSVRC 2014 classiﬁcation [26]. The best entry of each team is listed.
in ILSVRC 2013. We only consider single-network
performance for manageable comparisons.
Our best single network achieves 9.14% top-5 error
on the validation set. This is exactly the single-model
entry we submitted to ILSVRC 2014 [26]. The top-5
error is 9.08% on the testing set (ILSVRC 2014 has
the same training/validation/testing data as ILSVRC
2012). After combining eleven models, our team’s result ( 8.06%) is ranked #3 among all 38 teams attending
ILSVRC 2014 (Table 5). Since the advantages of SPPnet should be in general independent of architectures,
we expect that it will further improve the deeper and
larger convolutional architectures [33], [32].
3.2 Experiments on VOC 2007 Classiﬁcation
Our method can generate a full-view image representation. With the above networks pre-trained on
ImageNet, we extract these representations from the


Our method can generate a full-view image representation. With the above networks pre-trained on
ImageNet, we extract these representations from the
images in the target datasets and re-train SVM classiﬁers [38]. In the SVM training, we intentionally do
not use any data augmentation (ﬂip/multi-view). We
l2-normalize the features for SVM training.
The classiﬁcation task in Pascal VOC 2007 [22]
involves 9,963 images in 20 categories. 5,011 images
are for training, and the rest are for testing. The
performance is evaluated by mean Average Precision
(mAP). Table 6 summarizes the results.
8
(a) (b) (c) (d) (e)
model no SPP (ZF-5) SPP (ZF-5) SPP (ZF-5) SPP (ZF-5) SPP (Overfeat-7)
crop crop full full full
size 224224 224224 224- 392- 364conv 4 59.96 57.28 - - conv 5 66.34 65.43 - - pool 5=7(66) 69.14 68.76 70.82 71.67 76.09


fc6=8 74.86 75.55 77.32 78.78 81.58
fc7=9 75.90 76.45 78.39 80.10 82.44
Table 6: Classiﬁcation mAP in Pascal VOC 2007. For SPP-net, the pool 5=7layer uses the 66 pyramid level.
(a) (b) (c) (d)
model no SPP (ZF-5) SPP (ZF-5) SPP (ZF-5) SPP (Overfeat-7)
crop crop full full
size 224224 224224 224- 224conv 4 80.12 81.03 - conv 5 84.40 83.76 - pool 5=7(66) 87.98 87.60 89.46 91.46
SPP pool 5=7 - 89.47 91.44 93.42
fc6=8 87.86 88.54 89.50 91.83
fc7=9 85.30 86.10 87.08 90.00
Table 7: Classiﬁcation accuracy in Caltech101. For SPP-net, the pool 5=7layer uses the 66 pyramid level.
We start from a baseline in Table 6 (a). The model is


We start from a baseline in Table 6 (a). The model is
ZF-5 without SPP . To apply this model, we resize the
image so that its smaller dimension is 224, and crop
the center 224224 region. The SVM is trained via
the features of a layer. On this dataset, the deeper the
layer is, the better the result is. In Table 6 (b), we replace the no-SPP net with our SPP-net. As a ﬁrst-step
comparison, we still apply the SPP-net on the center
224224 crop. The results of the fc layers improve.
This gain is mainly due to multi-level pooling.
Table 6 (c) shows our results on full images, where
the images are resized so that the shorter side is 224.
We ﬁnd that the results are considerably improved
(78.39% vs. 76.45%). This is due to the full-image
representation that maintains the complete content.
Because the usage of our network does not depend
on scale, we resize the images so that the smaller
dimension is sand use the same network to extract
features. We ﬁnd that s= 392 gives the best results
(Table 6 (d)) based on the validation set. This is mainly
because the objects occupy smaller regions in VOC


(Table 6 (d)) based on the validation set. This is mainly
because the objects occupy smaller regions in VOC
2007 but larger regions in ImageNet, so the relative
object scales are different between the two sets. These
results indicate scale matters in the classiﬁcation tasks,
and SPP-net can partially address this “scale mismatch” issue.
In Table 6 (e) the network architecture is replaced
with our best model (Overfeat-7, multi-size trained),
and the mAP increases to 82.44% . Table 8 summarizes
our results and the comparisons with the state-of-theart methods. Among these methods, VQ [15], LCC
[18], and FK [19] are all based on spatial pyramids
matching, and [13], [4], [34], [6] are based on deepnetworks. In these results, Oquab et al.’s (77.7%) and
Chatﬁeld et al .’s (82.42%) are obtained by network
ﬁne-tuning and multi-view testing. Our result is comparable with the state of the art, using only a single
full-image representation and without ﬁne-tuning.
3.3 Experiments on Caltech101


full-image representation and without ﬁne-tuning.
3.3 Experiments on Caltech101
The Caltech101 dataset [21] contains 9,144 images in
102 categories (one background). We randomly sample 30 images per category for training and up to 50
images per category for testing. We repeat 10 random
splits and average the accuracy. Table 7 summarizes
our results.
There are some common observations in the Pascal
VOC 2007 and Caltech101 results: SPP-net is better
than the no-SPP net (Table 7 (b) vs. (a)), and the fullview representation is better than the crop ((c) vs. (b)).
But the results in Caltech101 have some differences
with Pascal VOC. The fully-connected layers are less
accurate, and the SPP layers are better. This is possibly
because the object categories in Caltech101 are less
related to those in ImageNet, and the deeper layers
are more category-specialized. Further, we ﬁnd that
the scale 224 has the best performance among the
scales we tested on this dataset. This is mainly because
the objects in Caltech101 also occupy large regions of
the images, as is the case of ImageNet.
Besides cropping, we also evaluate warping the
image to ﬁt the 224 224 size. This solution maintains


the images, as is the case of ImageNet.
Besides cropping, we also evaluate warping the
image to ﬁt the 224 224 size. This solution maintains
the complete content, but introduces distortion. On
the SPP (ZF-5) model, the accuracy is 89.91% using
the SPP layer as features - lower than 91.44% which
uses the same model on the undistorted full image.
9
method VOC 2007 Caltech101
VQ [15]y56.07 74.41 1.0
LLC [18]y57.66 76.95 0.4
FK [19]y61.69 77.78 0.6
DeCAF [13] - 86.91 0.7
Zeiler & Fergus [4] 75.90z86.50.5
Oquab et al. [34] 77.7 Chatﬁeld et al. [6] 82.42 88.540.3
ours 82.44 93.42 0.5
Table 8: Classiﬁcation results for Pascal VOC 2007
(mAP) and Caltech101 (accuracy).ynumbers reported
by [27].zour implementation as in Table 6 (a).
Table 8 summarizes our results compared with the


(mAP) and Caltech101 (accuracy).ynumbers reported
by [27].zour implementation as in Table 6 (a).
Table 8 summarizes our results compared with the
state-of-the-art methods on Caltech101. Our result
(93.42% ) exceeds the previous record (88.54%) by a
substantial margin (4.88%).
4 SPP- NET FOR OBJECT DETECTION
Deep networks have been used for object detection.
We brieﬂy review the recent state-of-the-art R-CNN
method [7]. R-CNN ﬁrst extracts about 2,000 candidate windows from each image via selective search
[20]. Then the image region in each window is warped
to a ﬁxed size (227 227). A pre-trained deep network
is used to extract the feature of each window. A
binary SVM classiﬁer is then trained on these features
for detection. R-CNN generates results of compelling
quality and substantially outperforms previous methods. However, because R-CNN repeatedly applies the
deep convolutional network to about 2,000 windows
per image, it is time-consuming. Feature extraction is
the major timing bottleneck in testing.
Our SPP-net can also be used for object detection.
We extract the feature maps from the entire image


the major timing bottleneck in testing.
Our SPP-net can also be used for object detection.
We extract the feature maps from the entire image
only once (possibly at multiple scales). Then we apply the spatial pyramid pooling on each candidate
window of the feature maps to pool a ﬁxed-length
representation of this window (see Figure 5). Because
the time-consuming convolutions are only applied
once, our method can run orders of magnitude faster.
Our method extracts window-wise features from
regions of the feature maps, while R-CNN extracts
directly from image regions. In previous works, the
Deformable Part Model (DPM) [23] extracts features
from windows in HOG [24] feature maps, and the
Selective Search (SS) method [20] extracts from windows in encoded SIFT feature maps. The Overfeat
detection method [5] also extracts from windows of
deep convolutional feature maps, but needs to predeﬁne the window size. On the contrary, our method
enables feature extraction in arbitrary windows from
the deep convolutional feature maps.
spatial pyramid
pooling layer
feature maps of conv 5
convolutional layersfixed-length representation
input imagewindow…...fully-connected layers (fc 6, fc 7)Figure 5: Pooling features from arbitrary windows


convolutional layersfixed-length representation
input imagewindow…...fully-connected layers (fc 6, fc 7)Figure 5: Pooling features from arbitrary windows
on feature maps. The feature maps are computed
from the entire image. The pooling is performed in
candidate windows.
4.1 Detection Algorithm
We use the “fast” mode of selective search [20] to
generate about 2,000 candidate windows per image.
Then we resize the image such that min(w;h) =s,
and extract the feature maps from the entire image.
We use the SPP-net model of ZF-5 (single-size trained)
for the time being. In each candidate window, we use
a 4-level spatial pyramid (1 1, 22, 33, 66, totally
50 bins) to pool the features. This generates a 12,800d (25650) representation for each window. These
representations are provided to the fully-connected
layers of the network. Then we train a binary linear
SVM classiﬁer for each category on these features.
Our implementation of the SVM training follows
[20], [7]. We use the ground-truth windows to generate the positive samples. The negative samples are
those overlapping a positive window by at most 30%


[20], [7]. We use the ground-truth windows to generate the positive samples. The negative samples are
those overlapping a positive window by at most 30%
(measured by the intersection-over-union (IoU) ratio).
Any negative sample is removed if it overlaps another
negative sample by more than 70%. We apply the standard hard negative mining [23] to train the SVM. This
step is iterated once. It takes less than 1 hour to train
SVMs for all 20 categories. In testing, the classiﬁer
is used to score the candidate windows. Then we use
non-maximum suppression [23] (threshold of 30%) on
the scored windows.
Our method can be improved by multi-scale feature
extraction. We resize the image such that min(w;h) =
s2S=f480;576;688;864;1200g, and compute the
feature maps of conv 5for each scale. One strategy of
combining the features from these scales is to pool
them channel-by-channel. But we empirically ﬁnd
that another strategy provides better results. For each
candidate window, we choose a single scale s2S
such that the scaled candidate window has a number
of pixels closest to 224 224. Then we only use the
feature maps extracted from this scale to compute


such that the scaled candidate window has a number
of pixels closest to 224 224. Then we only use the
feature maps extracted from this scale to compute
10
the feature of this window. If the pre-deﬁned scales
are dense enough and the window is approximately
square, our method is roughly equivalent to resizing
the window to 224 224 and then extracting features
from it. Nevertheless, our method only requires computing the feature maps once (at each scale) from the
entire image, regardless of the number of candidate
windows.
We also ﬁne-tune our pre-trained network, following [7]. Since our features are pooled from the conv 5
feature maps from windows of any sizes, for simplicity we only ﬁne-tune the fully-connected layers.
In this case, the data layer accepts the ﬁxed-length
pooled features after conv 5, and the fc 6;7layers and
a new 21-way (one extra negative category) fc 8layer
follow. The fc 8weights are initialized with a Gaussian
distribution of =0.01. We ﬁx all the learning rates to
1e-4 and then adjust to 1e-5 for all three layers. During


distribution of =0.01. We ﬁx all the learning rates to
1e-4 and then adjust to 1e-5 for all three layers. During
ﬁne-tuning, the positive samples are those overlapping with a ground-truth window by [0:5;1], and
the negative samples by [0:1;0:5). In each mini-batch,
25% of the samples are positive. We train 250k minibatches using the learning rate 1e-4, and then 50k
mini-batches using 1e-5. Because we only ﬁne-tune
the fc layers, the training is very fast and takes about
2 hours on the GPU (excluding pre-caching feature
maps which takes about 1 hour). Also following [7],
we use bounding box regression to post-process the
prediction windows. The features used for regression
are the pooled features from conv 5(as a counterpart
of the pool 5features used in [7]). The windows used
for the regression training are those overlapping with
a ground-truth window by at least 50%.
4.2 Detection Results
We evaluate our method on the detection task of the
Pascal VOC 2007 dataset. Table 9 shows our results
on various layers, by using 1-scale ( s=688) or 5-scale.


Pascal VOC 2007 dataset. Table 9 shows our results
on various layers, by using 1-scale ( s=688) or 5-scale.
Here the R-CNN results are as reported in [7] using
the AlexNet [3] with 5 conv layers. Using the pool 5
layers (in our case the pooled features), our result
(44.9%) is comparable with R-CNN’s result (44.2%).
But using the non-ﬁne-tuned fc 6layers, our results are
inferior. An explanation is that our fc layers are pretrained using image regions, while in the detection
case they are used on the feature map regions. The
feature map regions can have strong activations near
the window boundaries, while the image regions may
not. This difference of usages can be addressed by
ﬁne-tuning. Using the ﬁne-tuned fc layers (ftfc 6;7), our
results are comparable with or slightly better than
the ﬁne-tuned results of R-CNN. After bounding box
regression, our 5-scale result ( 59.2%) is 0.7% better
than R-CNN (58.5%), and our 1-scale result (58.0%)
is 0.5% worse.


than R-CNN (58.5%), and our 1-scale result (58.0%)
is 0.5% worse.
In Table 10 we further compare with R-CNN using the same pre-trained model of SPPnet (ZF-5). InSPP (1-sc) SPP (5-sc) R-CNN
(ZF-5) (ZF-5) (Alex-5)
pool 5 43.0 44.9 44.2
fc6 42.5 44.8 46.2
ftfc6 52.3 53.7 53.1
ftfc7 54.5 55.2 54.2
ftfc7bb 58.0 59.2 58.5
conv time (GPU) 0.053s 0.293s 8.96s
fc time (GPU) 0.089s 0.089s 0.07s
total time (GPU) 0.142s 0.382s 9.03s
speedup ( vs. RCNN) 64 24 Table 9: Detection results (mAP) on Pascal VOC 2007.
“ft” and “bb” denote ﬁne-tuning and bounding box
regression.
SPP (1-sc) SPP (5-sc) R-CNN


regression.
SPP (1-sc) SPP (5-sc) R-CNN
(ZF-5) (ZF-5) (ZF-5)
ftfc7 54.5 55.2 55.1
ftfc7bb 58.0 59.2 59.2
conv time (GPU) 0.053s 0.293s 14.37s
fc time (GPU) 0.089s 0.089s 0.089s
total time (GPU) 0.142s 0.382s 14.46s
speedup ( vs. RCNN) 102 38 Table 10: Detection results (mAP) on Pascal VOC 2007,
using the same pre-trained model of SPP (ZF-5).
this case, our method and R-CNN have comparable
averaged scores. The R-CNN result is boosted by
this pre-trained model. This is because of the better
architecture of ZF-5 than AlexNet, and also because
of the multi-level pooling of SPPnet (if using the noSPP ZF-5, the R-CNN result drops). Table 11 shows
the results for each category.
Table 11 also includes additional methods. Selective
Search (SS) [20] applies spatial pyramid matching on


the results for each category.
Table 11 also includes additional methods. Selective
Search (SS) [20] applies spatial pyramid matching on
SIFT feature maps. DPM [23] and Regionlet [39] are
based on HOG features [24]. The Regionlet method
improves to 46.1% [8] by combining various features including conv 5. DetectorNet [40] trains a deep
network that outputs pixel-wise object masks. This
method only needs to apply the deep network once
to the entire image, as is the case for our method. But
this method has lower mAP (30.5%).
4.3 Complexity and Running Time
Despite having comparable accuracy, our method is
much faster than R-CNN. The complexity of the convolutional feature computation in R-CNN is O(n
2272)with the window number n(2000). This complexity of our method is O(rs2)at a scales, where
ris the aspect ratio. Assume ris about 4/3. In the
single-scale version when s= 688 , this complexity is
11
method mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv


about 1/160 of R-CNN’s; in the 5-scale version, this
complexity is about 1/24 of R-CNN’s.
In Table 10, we provide a fair comparison on the
running time of the feature computation using the
same SPP (ZF-5) model . The implementation of RCNN is from the code published by the authors implemented in Caffe [35]. We also implement our feature
computation in Caffe . In Table 10 we evaluate the
average time of 100 random VOC images using GPU.
R-CNN takes 14.37s per image for convolutions, while
our 1-scale version takes only 0.053s per image. So
ours is 270faster than R-CNN. Our 5-scale version
takes 0.293s per image for convolutions, so is 49
faster than R-CNN. Our convolutional feature computation is so fast that the computational time of fc layers
takes a considerable portion. Table 10 shows that the
GPU time of computing the 4,096-d fc 7features is
0.089s per image. Considering both convolutional and
fully-connected features, our 1-scale version is 102
faster than R-CNN and is 1.2% inferior; our 5-scale


fully-connected features, our 1-scale version is 102
faster than R-CNN and is 1.2% inferior; our 5-scale
version is 38faster and has comparable results.
We also compares the running time in Table 9 where
R-CNN uses AlexNet [3] as is in the original paper
[7]. Our method is 24 to 64faster. Note that the
AlexNet [3] has the same number of ﬁlters as our ZF5 on each conv layer. The AlexNet is faster because
it uses splitting on some layers, which was designed
for two GPUs in [3].
We further achieve an efﬁcient full system with the
help of the recent window proposal method [25]. The
Selective Search (SS) proposal [20] takes about 1-2 seconds per image on a CPU. The method of EdgeBoxes
[25] only takes0.2s. Note that it is sufﬁcient to use
a fast proposal method during testing only. Using the
same model trained as above (using SS), we test proposals generated by EdgeBoxes only. The mAP is 52.8
without bounding box regression. This is reasonableconsidering that EdgeBoxes are not used for training.
Then we use both SS and EdgeBox as proposals in


without bounding box regression. This is reasonableconsidering that EdgeBoxes are not used for training.
Then we use both SS and EdgeBox as proposals in
the training stage, and adopt only EdgeBoxes in the
testing stage. The mAP is 56.3 without bounding box
regression, which is better than 55.2 (Table 10) due to
additional training samples. In this case, the overall
testing time is0.5s per image including all steps
(proposal and recognition). This makes our method
practical for real-world applications.
4.4 Model Combination for Detection
Model combination is an important strategy for boosting CNN-based classiﬁcation accuracy [3]. We propose a simple combination method for detection.
We pre-train another network in ImageNet, using
the same structure but different random initializations. Then we repeat the above detection algorithm.
Table 12 (SPP-net (2)) shows the results of this network. Its mAP is comparable with the ﬁrst network
(59.1% vs. 59.2%), and outperforms the ﬁrst network
in 11 categories.
Given the two models, we ﬁrst use either model
to score all candidate windows on the test image.
Then we perform non-maximum suppression on the
union of the two sets of candidate windows (with


to score all candidate windows on the test image.
Then we perform non-maximum suppression on the
union of the two sets of candidate windows (with
their scores). A more conﬁdent window given by
one method can suppress those less conﬁdent given
by the other method. After combination, the mAP
is boosted to 60.9% (Table 12). In 17 out of all 20
categories the combination performs better than either
individual model. This indicates that the two models
are complementary.
We further ﬁnd that the complementarity is mainly
because of the convolutional layers. We have tried to
combine two randomly initialized ﬁne-tuned results
of the same convolutional model, and found no gain.
12
4.5 ILSVRC 2014 Detection
The ILSVRC 2014 detection [26] task involves 200
categories. There are 450k/20k/40k images in the
training/validation/testing sets. We focus on the task
of the provided-data-only track (the 1000-category
CLS training data is not allowed to use).
There are three major differences between the detection (DET) and classiﬁcation (CLS) training datasets,
which greatly impacts the pre-training quality. First,


There are three major differences between the detection (DET) and classiﬁcation (CLS) training datasets,
which greatly impacts the pre-training quality. First,
the DET training data is merely 1/3 of the CLS
training data. This seems to be a fundamental challenge of the provided-data-only DET task. Second, the
category number of DET is 1/5 of CLS. To overcome
this problem, we harness the provided subcategory
labels2for pre-training. There are totally 499 nonoverlapping subcategories ( i.e., the leaf nodes in the
provided category hierarchy). So we pre-train a 499category network on the DET training set. Third, the
distributions of object scales are different between
DET/CLS training sets. The dominant object scale in
CLS is about 0.8 of the image length, but in DET is
about 0.5. To address the scale difference, we resize
each training image to min(w;h) = 400 (instead of
256), and randomly crop 224224views for training.
A crop is only used when it overlaps with a ground
truth object by at least 50%.
We verify the effect of pre-training on Pascal VOC
2007. For a CLS-pre-training baseline, we consider


truth object by at least 50%.
We verify the effect of pre-training on Pascal VOC
2007. For a CLS-pre-training baseline, we consider
the pool 5features (mAP 43.0% in Table 9). Replaced
with a 200-category network pre-trained on DET, the
mAP signiﬁcantly drops to 32.7%. A 499-category
pre-trained network improves the result to 35.9%.
Interestingly, even if the amount of training data
do not increase, training a network of more categories boosts the feature quality. Finally, training with
min(w;h) = 400 instead of 256 further improves the
mAP to 37.8%. Even so, we see that there is still a
considerable gap to the CLS-pre-training result. This
indicates the importance of big data to deep learning.
For ILSVRC 2014, we train a 499-category Overfeat7 SPP-net. The remaining steps are similar to the
VOC 2007 case. Following [7], we use the validation
set to generate the positive/negative samples, with
windows proposed by the selective search fast mode.
The training set only contributes positive samples
using the ground truth windows. We ﬁne-tune the fc
layers and then train the SVMs using the samples in


The training set only contributes positive samples
using the ground truth windows. We ﬁne-tune the fc
layers and then train the SVMs using the samples in
both validation and training sets. The bounding box
regression is trained on the validation set.
Our single model leads to 31.84% mAP in the
ILSVRC 2014 testing set [26]. We combine six similar
models using the strategy introduced in this paper.
The mAP is 35.11% in the testing set [26]. This result
ranks #2 in the provided-data-only track of ILSVRC
2014 (Table 13) [26]. The winning result is 37.21% from
2. Using the provided subcategory labels is allowed, as is explicitly stated in the competition introduction.rank team mAP
1 NUS 37.21
2 ours 35.11
3 UvA 32.02
- (our single-model) (31.84)
4 Southeast-CASIA 30.47
5 1-HKUST 28.86
6 CASIA CRIPAC 2 28.61
Table 13: The competition results of ILSVRC 2014
detection (provided-data-only track) [26]. The best
entry of each team is listed.
NUS, which uses contextual information.
Our system still shows great advantages on speed


entry of each team is listed.
NUS, which uses contextual information.
Our system still shows great advantages on speed
for this dataset. It takes our single model 0.6 seconds
(0.5 for conv, 0.1 for fc, excluding proposals) per testing image on a GPU extracting convolutional features
from all 5 scales. Using the same model, it takes 32
seconds per image in the way of RCNN. For the 40k
testing images, our method requires 8 GPU hours to
compute convolutional features, while RCNN would
require 15 GPUdays.
5 C ONCLUSION
SPP is a ﬂexible solution for handling different scales,
sizes, and aspect ratios. These issues are important in
visual recognition, but received little consideration in
the context of deep networks. We have suggested a solution to train a deep network with a spatial pyramid
pooling layer. The resulting SPP-net shows outstanding accuracy in classiﬁcation/detection tasks and
greatly accelerates DNN-based detection. Our studies
also show that many time-proven techniques/insights
in computer vision can still play important roles in
deep-networks-based recognition.
APPENDIX A
In the appendix, we describe some implementation
details:
Mean Subtraction.
The 224224 cropped training/testing images are


APPENDIX A
In the appendix, we describe some implementation
details:
Mean Subtraction.
The 224224 cropped training/testing images are
often pre-processed by subtracting the per-pixel mean
[3]. When input images are in any sizes, the ﬁxedsize mean image is not directly applicable. In the
ImageNet dataset, we warp the 224 224 mean image
to the desired size and then subtract it. In Pascal VOC
2007 and Caltech101, we use the constant mean (128)
in all the experiments.
Implementation of Pooling Bins.
We use the following implementation to handle all
bins when applying the network. Denote the width
and height of the conv 5feature maps (can be the
full image or a window) as wandh. For a pyramid level with nnbins, the (i;j)-th bin is in the
range of [bi1
nwc;di
nwe][bj1
nhc;dj
nhe]. Intuitively,
13
bottle: 0. 24person:1.20
sheep:1.52
chair:0.21
diningtable:0.78person:1.16pers on:1.05


sheep:1.52
chair:0.21
diningtable:0.78person:1.16pers on:1.05
pottedplant:0.21chair:4.79pottedplant:0.73
c hair: 0. 33diningtable: 0.34
chair:0.89
bus:0.56
car:3.24
ca r:3.4 5pers on:1.52
train:0.31
train:1.62
pottedplant:0.33
pottedplant:0.78
sofa:0.55tvmonitor:1.77
aeroplane: 0.45
aeroplane:1.40aeroplane:1.01
aeroplane:0.94aeroplane:0.93
aeroplane:0.91aeroplane:0.79
aeroplane:0.57 aeroplane:0.54
person:0.93
person:0.68
horse:1.73pers on:1.91
boat:0.60pers on:3.20
car:2.52
bus:2.01
car:0.93per son:4.16person:0.79
pers on:0.32
hors e:1.68
hors e:0.61
horse:1.29


pers on:0.32
hors e:1.68
hors e:0.61
horse:1.29
per son:1.23
c hair:0.87pers on:1.79
per son:0.91
sofa:0.22person:0.85
sofa:0.58
c ow:2.36
cow:1.88co w:1 .86
cow:1.82cow:1.39 cow:1.31
ca t:0.5 2person:1.02
person:0.40
bicy cle:2.85bicy cle:2.71
bicycle:2.04bicy cle:0.67person:3.35
person:2.39
pers on:2.11
pers on:0.27
person:0.22bus:1.42
person:3.29person:1.18
bottle:1.15pottedplant:0.81
s heep:1.81s heep:1.17
sheep:0.81
bird:0.24pott edplant:0.35 pot tedplant: 0. 20
car:1.31person:1.60
person:0.62
dog:0.37person:0.38
dog:0.99person:1.48


car:1.31person:1.60
person:0.62
dog:0.37person:0.38
dog:0.99person:1.48
pers on:0.22
cow:0.80person:3.29
person:2.69pers on:2.42
person:1.05
person:0.92pers on:0.76
bird:1.39
bird:0.84
bottle:1.20diningtable:0.96pers on:1.53pers on:1.52
pers on:0.73
car:0.12
car:0.11
car:0.04car:0.03
car:3.98car:1.95car:1.39
car:0.50
bird:1.47
sofa:0.41 person:2.15pers on:0 .86tvmonitor:2.24
motorbike:1.11motorbike:0.74person:1.36
pers on:1.10
Figure 6: Example detection results of “SPP-net ftfc 7bb” on the Pascal VOC 2007 testing set (59.2% mAP).
All windows with scores >0 are shown. The predicted category/score are marked. The window color is


All windows with scores >0 are shown. The predicted category/score are marked. The window color is
associated with the predicted category. These images are manually selected because we ﬁnd them impressive.
Visit our project website to see all 4,952 detection results in the testing set.
if rounding is needed, we take the ﬂoor operation on
the left/top boundary and ceiling on the right/bottom
boundary.
Mapping a Window to Feature Maps.
In the detection algorithm (and multi-view testing
on feature maps), a window is given in the image
domain, and we use it to crop the convolutional feature maps ( e.g., conv 5) which have been sub-sampled
several times. So we need to align the window on the
feature maps.
In our implementation, we project the corner point
of a window onto a pixel in the feature maps, such
that this corner point in the image domain is closest
to the center of the receptive ﬁeld of that feature mappixel. The mapping is complicated by the padding
of all convolutional and pooling layers. To simplify
the implementation, during deployment we pad bp=2c
pixels for a layer with a ﬁlter size of p. As such, for
a response centered at (x0;y0), its effective receptive


pixels for a layer with a ﬁlter size of p. As such, for
a response centered at (x0;y0), its effective receptive
ﬁeld in the image domain is centered at (x;y) =
(Sx0;Sy0)whereSis the product of all previous
strides. In our models, S= 16 for ZF-5 on conv 5,
andS= 12 for Overfeat-5/7 on conv 5=7. Given a
window in the image domain, we project the left (top)
boundary by: x0=bx=Sc+ 1 and the right (bottom)
boundaryx0=dx=Se1. If the padding is not bp=2c,
we need to add a proper offset to x.
14
REFERENCES
[1] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard,
W. Hubbard, and L. D. Jackel, “Backpropagation applied to
handwritten zip code recognition,” Neural computation , 1989.


W. Hubbard, and L. D. Jackel, “Backpropagation applied to
handwritten zip code recognition,” Neural computation , 1989.
[2] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei, “Imagenet: A large-scale hierarchical image database,” in
CVPR , 2009.
[3] A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classiﬁcation with deep convolutional neural networks,” in NIPS ,
2012.
[4] M. D. Zeiler and R. Fergus, “Visualizing and understanding
convolutional neural networks,” arXiv:1311.2901 , 2013.
[5] P . Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y. LeCun, “Overfeat: Integrated recognition, localization
and detection using convolutional networks,” arXiv:1312.6229 ,
2013.
[6] A. V . K. Chatﬁeld, K. Simonyan and A. Zisserman, “Return of


2013.
[6] A. V . K. Chatﬁeld, K. Simonyan and A. Zisserman, “Return of
the devil in the details: Delving deep into convolutional nets,”
inArXiv:1405.3531 , 2014.
[7] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature
hierarchies for accurate object detection and semantic segmentation,” in CVPR , 2014.
[8] W. Y. Zou, X. Wang, M. Sun, and Y. Lin, “Generic object detection with dense neural patterns and regionlets,” in
ArXiv:1404.4316 , 2014.
[9] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “Cnn
features off-the-shelf: An astounding baseline for recogniton,”
inCVPR 2014, DeepVision Workshop , 2014.
[10] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface:
Closing the gap to human-level performance in face veriﬁcation,” in CVPR , 2014.


Closing the gap to human-level performance in face veriﬁcation,” in CVPR , 2014.
[11] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdevr,
“Panda: Pose aligned networks for deep attribute modeling,”
inCVPR , 2014.
[12] Y. Gong, L. Wang, R. Guo, and S. Lazebnik, “Multi-scale
orderless pooling of deep convolutional activation features,”
inArXiv:1403.1840 , 2014.
[13] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng,
and T. Darrell, “Decaf: A deep convolutional activation feature
for generic visual recognition,” arXiv:1310.1531 , 2013.
[14] K. Grauman and T. Darrell, “The pyramid match kernel:
Discriminative classiﬁcation with sets of image features,” in
ICCV , 2005.
[15] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recognizing natural scene


[15] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recognizing natural scene
categories,” in CVPR , 2006.
[16] J. Sivic and A. Zisserman, “Video google: a text retrieval
approach to object matching in videos,” in ICCV , 2003.
[17] J. Yang, K. Yu, Y. Gong, and T. Huang, “Linear spatial pyramid
matching using sparse coding for image classiﬁcation,” in
CVPR , 2009.
[18] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, “Localityconstrained linear coding for image classiﬁcation,” in CVPR ,
2010.
[19] F. Perronnin, J. S ´anchez, and T. Mensink, “Improving the ﬁsher
kernel for large-scale image classiﬁcation,” in ECCV , 2010.
[20] K. E. van de Sande, J. R. Uijlings, T. Gevers, and A. W. Smeulders, “Segmentation as selective search for object recognition,”


inICCV , 2011.
[21] L. Fei-Fei, R. Fergus, and P . Perona, “Learning generative
visual models from few training examples: An incremental
bayesian approach tested on 101 object categories,” CVIU ,
2007.
[22] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and
A. Zisserman, “The PASCAL Visual Object Classes Challenge
2007 (VOC2007) Results,” 2007.
[23] P . F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained partbased models,” P AMI , 2010.
[24] N. Dalal and B. Triggs, “Histograms of oriented gradients for
human detection,” in CVPR , 2005.
[25] C. L. Zitnick and P . Doll ´ar, “Edge boxes: Locating object
proposals from edges,” in ECCV , 2014.
[26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,


[26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein
et al. , “Imagenet large scale visual recognition challenge,”
arXiv:1409.0575 , 2014.
[27] K. Chatﬁeld, V . Lempitsky, A. Vedaldi, and A. Zisserman, “The
devil is in the details: an evaluation of recent feature encoding
methods,” in BMVC , 2011.
[28] A. Coates and A. Ng, “The importance of encoding versus
training with sparse coding and vector quantization,” in ICML ,
2011.
[29] D. G. Lowe, “Distinctive image features from scale-invariant
keypoints,” IJCV , 2004.
[30] J. C. van Gemert, J.-M. Geusebroek, C. J. Veenman, and A. W.
Smeulders, “Kernel codebooks for scene categorization,” in


Smeulders, “Kernel codebooks for scene categorization,” in
ECCV , 2008.[31] M. Lin, Q. Chen, and S. Yan, “Network in network,”
arXiv:1312.4400 , 2013.
[32] C. Szegedy, W. Liu, Y. Jia, P . Sermanet, S. Reed, D. Anguelov,
D. Erhan, V . Vanhoucke, and A. Rabinovich, “Going deeper
with convolutions,” arXiv:1409.4842 , 2014.
[33] K. Simonyan and A. Zisserman, “Very deep convolutional
networks for large-scale image recognition,” arXiv:1409.1556 ,
2014.
[34] M. Oquab, L. Bottou, I. Laptev, J. Sivic et al. , “Learning and
transferring mid-level image representations using convolutional neural networks,” in CVPR , 2014.
[35] Y. Jia, “Caffe: An open source convolutional architecture
for fast feature embedding,” http://caffe.berkeleyvision.org/,
2013.


for fast feature embedding,” http://caffe.berkeleyvision.org/,
2013.
[36] A. G. Howard, “Some improvements on deep convolutional
neural network based image classiﬁcation,” ArXiv:1312.5402 ,
2013.
[37] H. Jegou, F. Perronnin, M. Douze, J. Sanchez, P . Perez, and
C. Schmid, “Aggregating local image descriptors into compact
codes,” TP AMI , vol. 34, no. 9, pp. 1704–1716, 2012.
[38] C.-C. Chang and C.-J. Lin, “Libsvm: a library for support
vector machines,” ACM Transactions on Intelligent Systems and
Technology (TIST) , 2011.
[39] X. Wang, M. Yang, S. Zhu, and Y. Lin, “Regionlets for generic
object detection,” in ICCV , 2013.
[40] C. Szegedy, A. Toshev, and D. Erhan, “Deep neural networks
for object detection,” in NIPS , 2013.
CHANGELOG
arXiv v1 . Initial technical report for ECCV 2014 paper.


for object detection,” in NIPS , 2013.
CHANGELOG
arXiv v1 . Initial technical report for ECCV 2014 paper.
arXiv v2 . Submitted version for TPAMI. Includes extra
experiments of SPP on various architectures. Includes
details for ILSVRC 2014.
arXiv v3 . Accepted version for TPAMI. Includes comparisons with R-CNN using the same architecture.
Includes detection experiments using EdgeBoxes.
arXiv v4 . Revised “Mapping a Window to Feature
Maps” in Appendix for easier implementation.


Accepted as a workshop contribution at ICLR 2015
LEARNING DEEPSTRUCTURED MODELS
Liang-Chieh Chen
Department of Computer Science, UCLA, lcchen@cs.ucla.edu
Alexander G. Schwing
Department of Computer Science, University of Toronto, aschwing@cs.toronto.edu
Alan L. Yuille
Department of Statistics, UCLA, yuille@stat.ucla.edu
Raquel Urtasun
Department of Computer Science, University of Toronto, urtasun@cs.toronto.edu
ABSTRACT
Many problems in real-world applications involve predicting several random variables which are statistically related. Markov random ﬁelds (MRFs) are a great
mathematical tool to encode such relationships. The goal of this paper is to combine MRFs with deep learning algorithms to estimate complex representations
while taking into account the dependencies between the output random variables.
Towards this goal, we propose a training algorithm that is able to learn structured
models jointly with deep features that form the MRF potentials. Our approach is
efﬁcient as it blends learning and inference and makes use of GPU acceleration.
We demonstrate the effectiveness of our algorithm in the tasks of predicting words
from noisy images, as well as multi-class classiﬁcation of Flickr photographs. We

Learning Deep Structured Models

Summary:
Many problems in real-world applications involve predicting several random
variables which are statistically related. Markov random fields (MRFs) are a
great mathematical tool to encode such relationships. The goal of this paper is
to combine MRFs with deep learning algorithms to estimate complex
representations while taking into account the dependencies between the output
random variables. Towards this goal, we propose a training algorithm that is
able to learn structured models jointly with deep features that form the MRF
potentials. Our approach is efficient as it blends learning and inference and
makes use of GPU acceleration. We demonstrate the effectiveness of our
algorithm in the tasks of predicting words from noisy images, as well as
multi-class classification of Flickr photographs. We show that joint learning
of the deep features and the MRF parameters results in significant performance
gains.

Liang-Chieh Chen
Alexander G. Schwing
Alan L. Yuille
Raquel Urtasun

11 pages including reference


We demonstrate the effectiveness of our algorithm in the tasks of predicting words
from noisy images, as well as multi-class classiﬁcation of Flickr photographs. We
show that joint learning of the deep features and the MRF parameters results in
signiﬁcant performance gains.
1 I NTRODUCTION
Deep learning algorithms attempt to model high-level abstractions of the data using architectures composed of multiple non-linear transformations. A multiplicity of variants have been proposed (Hinton et al., 1984; LeCun et al., 1998; Hinton & Salakhutdinov, 2006; Bengio et al., 2007;
Salakhutdinov & Hinton, 2012; Zeiler & Fergus, 2014) and shown to be extremely successful in a
wide variety of applications including object detection, speech recognition as well as natural language processing (Lee et al., 2009; Socher et al., 2012; Jia, 2013; Krizhevsky et al., 2013; Eigen
et al., 2014). Recently, state-of-the-art results have been achieved in many computer vision tasks,
outperforming competitive methods by a large margin (Krizhevsky et al., 2013; Girshick et al.,
2014).
Deep neural networks can, however, be even more powerful when combined with graphical models

Learning Deep Structured Models

11 pages including reference


2014).
Deep neural networks can, however, be even more powerful when combined with graphical models
in order to capture the statistical dependencies between the variables of interest. For example, Deng
et al. (2014) exploit mutual exclusion, overlapping and subsumption properties of class labels in order to better predict in large scale classiﬁcation tasks. In pose estimation, more accurate predictions
can be obtained when encoding the spatial relationships between joint locations (Tompson et al.,
2014).
It is, however, an open problem how to develop scalable deep learning algorithms that can learn
higher-order knowledge taking into account the output variable’s dependencies. Existing approaches
often rely on a two-step process (Nowozin et al., 2011; Xu et al., 2014) where a non-linear classiﬁer
that employs deep features is trained ﬁrst, and its output is used to generate potentials for the structured predictor. This piece-wise training is, however, suboptimal as the deep features are learned
The ﬁrst two authors contributed equally to this work.
1arXiv:1407.2538v3  [cs.LG]  27 Apr 2015
Accepted as a workshop contribution at ICLR 2015

Learning Deep Structured Models

11 pages including reference


1arXiv:1407.2538v3  [cs.LG]  27 Apr 2015
Accepted as a workshop contribution at ICLR 2015
while ignoring the dependencies between the variables of interest, e.g., independently learned segmentation and detection features (Hariharan et al., 2014) might be focusing on predicting the same
examples correctly. But when learned jointly they can improve their predictive power by exploiting
complementary information to ﬁx additional mistakes.
In this paper we extend deep learning algorithms to learn complex representations taking into account the dependencies between the output random variables. Towards this goal, we propose a
learning algorithm that is able to learn structured models with arbitrary graphs jointly with deep
features that form Markov random ﬁeld (MRF) potentials. Our approach is efﬁcient as it blends
learning and inference resulting in a single loop algorithm which makes use of GPU acceleration.
We demonstrate the effectiveness of our method in the tasks of predicting words from noisy images,
and multi-class classiﬁcation of Flickr photographs. We show that joint learning of deep features
and MRF parameters results in big performance gains.
2 L EARNING DEEPSTRUCTURED MODELS

Learning Deep Structured Models

11 pages including reference


and MRF parameters results in big performance gains.
2 L EARNING DEEPSTRUCTURED MODELS
In this section we investigate how to learn ‘deep features’ that take into account the dependencies between the output variables. Let y2Y be the set of random variables y= (y1;:::;yN)
that we are interested in predicting. In this work we assume the space of valid conﬁgurations to
be a product space, i.e.,Y=QN
i=1Yi, and the domain of each individual variable yito be discrete, i.e.,Yi=f1;:::;jYijg. Given input data x2X and parameters w2RAof the function
F(x;y;w) :XY RA!R, inference amounts to ﬁnding the highest scoring conﬁguration
y= arg max yF(x;y;w):Note that ifFis a deep network and there are no connections between
the output variables to be predicted, inference corresponds to a forward pass to evaluate the function,
followed by independently ﬁnding the largest response for each variable. This can be interpreted as
inference in a graphical model with only unary potentials. However, for arbitrary graphical models

Learning Deep Structured Models

11 pages including reference


inference in a graphical model with only unary potentials. However, for arbitrary graphical models
it is NP-hard to ﬁnd the maximizing conﬁguration ysince the inference program generally requires
a search over a space of sizeQN
i=1jYij. Note also that log-linear models are a special case of this
program, with F(x;y;w) =w>(x;y)and(x;y)denoting a feature vector computed using the
input-output pair (x;y).
In this work, we consider the general setting where F(x;y;w)is an arbitrary scalar-valued function ofwand(x;y). In our experiments Fis a function composition of non-linear base mappings
like convolutions, rectiﬁcations, pooling etc. We let the probability of an arbitrary conﬁguration ^y
be given by the annealed soft-max p(x;y)(^yjw;) =1
Z(x;w)exp(F(x;^y;w))1=:HerebyZ(x;w)
refers to the partition function, normalizing the distribution p(x;y)to lie within the probability simplexviaZ(x;w) =P

Learning Deep Structured Models

11 pages including reference


refers to the partition function, normalizing the distribution p(x;y)to lie within the probability simplexviaZ(x;w) =P
^y2Yexp(F(x;^y;w))1=. The annealing/temperature parameter 0is
used to adjust the uniformity of the distribution. We consider general graphical models where the
computation of Z(x;w)is #P-hard.
2.1 L EARNING VIA GRADIENT DESCENT
During learning, given a training set Dof input-output pairs (x;y)2D, we are interested in ﬁnding
the parameters wof the model. We do so by maximizing the data likelihood, i.e., minimizing the
negative log-likelihood lnQ
(x;y)2Dp(x;y)(yjw;)which yields
min
wX
(x;y)2D(lnZ(x;w)F(x;y;w)): (1)
Note that this is equivalent to maximizing the cross-entropy between a target distributionp(x;y);tg(^y) =(^y=y)placing all its mass on the groundtruth label,

Learning Deep Structured Models

11 pages including reference


and the model distribution p(x;y)(^yjw;). Hence Eq. (1) is equivalently obtained by
maxwP
(x;y);^y2Yp(x;y);tg(^y) lnp(x;y)(^yjw;). It is easily possible to incorporate more general target distributions into Eq. (1). Note also that = 0recovers the structured hinge loss objective.
Minimizing Eq. (1) w.r.t. wrequires computation of the gradient@
@wP
(x;y)2Dlnp(x;y)(yjw;),
which is given by a transformed difference between the distributions of the model p(x;y)(^yjw;)and
2
Accepted as a workshop contribution at ICLR 2015
Repeat until stopping criteria
1. Forward pass to compute F(x;^y;w)
2. Compute p(x;y)(^yjw;)
3. Backward pass via chain rule to obtain gradient
4. Update parameters w
Figure 1: Gradient descent algorithm for learning deep structured models.
the targetp(x;y);tg(^y):
X
(x;y)2DX
^y2Y@

Learning Deep Structured Models

11 pages including reference


the targetp(x;y);tg(^y):
X
(x;y)2DX
^y2Y@
@wF(x;^y;w)
p(x;y)(^yjw;)p(x;y);tg(^y)
: (2)
A gradient descent algorithm for minimizing Eq. (1) will iterate between the following steps: (i) For
a givenwevaluate the function F, (ii) compute the model distribution p(x;y)(^yjw;), (iii) propagate
the difference between the model and target distribution using a backward pass (resembling the chain
rule for composite functions) and (iv) update the parameters w. This is summarized in Fig. 1.
2.2 A PPROXIMATE LEARNING
Note that for general graphical models the exact computation of p(x;y)(^yjw;)is not possible. As a
consequence it is intractable to compute the exact gradient of the cost-function given in Eq. (2) and
one has to resort to approximate solutions.
Inspired by approximations used for log-linear models, we make use of the following identity (Wainwright & Jordan, 2008; Koller & Friedman, 2009):

Learning Deep Structured Models

11 pages including reference


Inspired by approximations used for log-linear models, we make use of the following identity (Wainwright & Jordan, 2008; Koller & Friedman, 2009):
lnZ(x;w) = max
p(x;y)(^y)2E[F(x;^y;w)] +H(p(x;y)); (3)
where Edenotes an expectation over p(x;y)(^y)andHrefers to the entropy.
For most applications, F(x;y;w)decomposes into a sum of functions, each depending on a local
subset of variables yr,i.e.,F(x;y;w) =P
r2Rfr(x;yr;w):Herebyris a restriction of the variable
tupley= (y1;:::;yN)to the subset rf1;:::;Ng,i.e.,yr= (yi)i2r. All subsets rrequired to
compute the model function Fare summarized in the set R.
Plugging this decomposition into Eq. (3), we equivalently get the log-partition function
lnZ(x;w)viamaxp(x;y)(^y)2P

Learning Deep Structured Models

11 pages including reference


lnZ(x;w)viamaxp(x;y)(^y)2P
r;^yrp(x;y);r(^yr)fr(x;^yr;w) +H(p(x;y));where we use
marginalsp(x;y);r(^yr) =P
ynyrp(x;y)(y).
Despite the assumed locality of the scoring function, the learning task remains computationally
challenging since the entropy H(p(x;y))can only be computed exactly for a very small set of models,
e.g., models for which the joint distribution p(x;y)(y)is equivalently described by low tree-width
models. In addition, the marginalization constraints are exponential in size.
To deal with both issues a common solution in log-linear models is to approximate the true marginals
p(x;y);rwith local beliefs b(x;y);rthat are not required to fulﬁll marginalization constraints globally,
but only locally (Wainwright & Jordan, 2008). That means marginals b(x;y);rare not required to arise
from a common joint distribution p(x;y). In addition, we approximate the entropy via the fractional

Learning Deep Structured Models

11 pages including reference


from a common joint distribution p(x;y). In addition, we approximate the entropy via the fractional
entropy (Wiegerinck & Heskes, 2003), i.e.,H(p(x;y))P
rcrH(b(x;y);r). Counting numbers cr
are employed to weight the marginal entropies. Putting all this together, we obtain the following
approximation for lnZ(x;w):
max
b(x;y)2C(x;y)X
r;^yrb(x;y);r(^yr)fr(x;^yr;w) +X
rcrH(b(x;y);r): (4)
Hereby beliefs are constrained to the local polytope
C(x;y)=(8r b (x;y);r2
8r;^yr;p2P(r)P
^ypn^yrb(x;y);p(^yp) =b(x;y);r(^yr);
3
Accepted as a workshop contribution at ICLR 2015
Let~cr;p=cp=(cr+P
p02P(r)cp0). Repeat until stopping criteria

Learning Deep Structured Models

11 pages including reference


3
Accepted as a workshop contribution at ICLR 2015
Let~cr;p=cp=(cr+P
p02P(r)cp0). Repeat until stopping criteria
1. Forward pass to compute fr(x;^yr;w)8(x;y);r;yr
2. Get beliefs b(x;y);r/exp^fr(x;^yr;w;)
crby iterating over r:8(x;y);p2P(r);^yr
(x;y);p!r(^yr) =cplnX
^ypn^yrexpfp(x;^yp;w)P
p02P(p)(x;y);p!p0(^yp) +P
r02C(p)nr(x;y);r0!p(^yr0)
cp
(x;y);r!p(^yr)/~cr;p0
@fr(x;^yr;w) +X
c2C(r)(x;y);c!r(^yc) +X
p2P(r)(x;y);p!r(^yr)1
A(x;y);p!r(^yr)

Learning Deep Structured Models

11 pages including reference


p2P(r)(x;y);p!r(^yr)1
A(x;y);p!r(^yr)
3. Backward pass via chain-rule for gradient g=P
(x;y);r;^yrb(x;y);r(^yr)rwfr(x;^yr;w)rwF(w)
4. Update parameters wusing stepsize viaw wg
Figure 2: Efﬁcient learning algorithm that blends learning and inference.
withP(r)the set of parents of region r,i.e.,P(r)fp2R :rpg, which subsumes those
regions for which we want the marginalization constraint to hold. Conversely, we deﬁne the set of
children asC(r) =fc2R:r2P(c)g.
We can thus rewrite the learning problem by plugging the approximations derived in Eq. (4) into
Eq. (1). This gives rise to the new approximated learning program
min
wX
(x;y)2D0
@max
b(x;y)2C(x;y)8
<
:X

Learning Deep Structured Models

11 pages including reference


min
wX
(x;y)2D0
@max
b(x;y)2C(x;y)8
<
:X
r;^yrb(x;y);r(^yr)fr(x;^yr;w) +X
rcrH(b(x;y);r)9
=
;F(x;y;w)1
A:(5)
To iteratively update the parameters for the non-smooth approximated cost function given in Eq. (5)
we require a sub-gradient w.r.t. w, which in turn requires to solve the maximization w.r.t. the beliefs
bexactly. This is a non-trivial task in itself as inference in general graphical models is NP-hard.
Iterative message passing algorithms (Pearl, 1988; Yedidia et al., 2005; Wainwright et al., 2005;
Weiss et al., 2007; Meltzer et al., 2009) are typically employed. Importantly, note that combining
the procedure outlined in Fig. 1 with iterative message passing to approximate p(x;y)(^yjw;)results
in a double-loop algorithm which would be slow for many graphical models of interest.

Learning Deep Structured Models

11 pages including reference


in a double-loop algorithm which would be slow for many graphical models of interest.
2.3 E FFICIENT APPROXIMATE LEARNING BY BLENDING LEARNING AND INFERENCE
In this section we propose a more efﬁcient algorithm that is based on the principle of blending
learning ( i.e., parameter updates) and inference. Thus we are interested in only performing a single
message passing iteration before updating the parameters w. Note that simply reducing the number
of iterations is generally not an option as the obtained beliefs b(x;y);rare by no means accurate.
However, assuming all counting numbers crto be positive, we can derive an algorithm that is able
to interleave minimization w.r.t. wand maximization of the beliefs b. Such a procedure is more
efﬁcient as we are able to update the parameters wmuch more frequently.
To interleave both programs we convert maximization of the beliefs into a minimization by employing the dual program as detailed in the following claim. This is possible since the maximization
problem is concave in b(x;y)if8r,cr0.
Claim 1 Assumecr08rand letF(w) =P
(x;y)2DF(x;y;w)denote the sum of empirical

Learning Deep Structured Models

11 pages including reference


Claim 1 Assumecr08rand letF(w) =P
(x;y)2DF(x;y;w)denote the sum of empirical
function observations. Let (x;y);r!p(^yr)be the Lagrange multipliers for each marginalization
constraintP
^ypn^yrb(x;y);p(^yp) =b(x;y);r(^yr)within the polytope C(x;y). Then the approximated
general structured prediction task shown in Eq. (5)is equivalent to
min
w;X
(x;y);rcrlnX
^yrexp^fr(x;^yr;w;)
crF(w); (6)
4
Accepted as a workshop contribution at ICLR 2015
banal julep resty
drein yojan mothy
Figure 3: Samples from the Word50 dataset. High degree of rotation, scaling and translation.
where we employed the re-parameterization score ^fr(x;^yr;w;) =fr(x;^yr;w) +P
c2C(r)(x;y);c!r(^yc)P

Learning Deep Structured Models

11 pages including reference


c2C(r)(x;y);c!r(^yc)P
p2P(r)(x;y);r!p(^yr).
Proof: To obtain the dual of the maximization w.r.t. b(x;y)we utilize its Lagrangian L(x;y)=P
r;^yrb(x;y);r(^yr)^fr(x;^yr;w;)+P
rcrH(b(x;y);r):Maximization of the Lagrangian w.r.t. the primal variables bis possible by employing the relationship stated in Eq. (3) locally 8r. We then obtain
the dual function being the ﬁrst term in Eq. (6). For strict convexity, i.e.,cr>0, we reconstruct the beliefs to be proportional to the exponentiated, loss-augmented re-parameterization score
b(x;y);r/exp^fr(x;^yr;w;)
cr:Forcr= 0the beliefs correspond to a uniform distribution over the set
of maximizers of the loss-augmented re-parameterization score ^fr(x;^yr;w;).

Learning Deep Structured Models

11 pages including reference


of maximizers of the loss-augmented re-parameterization score ^fr(x;^yr;w;).
It is important to note that by applying duality we managed to convert the min-max task in Eq. (5)
into a single minimization as shown in Eq. (6). Performing block coordinate descent updates to
minimize Eq. (6), we are therefore able to interleave both, updating the weights ( i.e., learning) and
the messages ( i.e., inference). This results in a more efﬁcient algorithm, as inference does not have
to be run until convergence. Even a single update of the messages sufﬁces. We note that this is
possible only if cr08r. Strictly speaking, we require concavity only within the set of feasible
beliefsC(x;y). However, for simplicity we neglect this extension in the following.
Fig. 2 summarizes our efﬁcient deep structured prediction algorithm which iterates between the
following steps. Given parameters wwe perform a standard forward pass to compute fr(x;^yr;w)
for all regions. We then iterate through all regions rand use block-coordinate descent to ﬁnd the

Learning Deep Structured Models

11 pages including reference


for all regions. We then iterate through all regions rand use block-coordinate descent to ﬁnd the
globally optimal value of Eq. (6) w.r.t. (x;y);r!p(^yr)8(x;y);^yr;p2P(r). This can be done in
closed form and therefore is computed very efﬁciently. We refer the reader to Schwing (2013) for
a derivation in the log-linear setting. We then compute the gradient using a standard backward pass
before we update the parameters by performing a step of size along the negative gradient.
2.4 I MPLEMENTATION DETAILS
We implemented the general algorithm presented in Fig. 2 in C++ as a library for Linux, Windows
and OS X platforms. It supports usage of the GPU for the forward and backward pass using both,
standard linear algebra packages and manually tuned GPU-kernels. In addition to standard gradient descent, we allow speciﬁcation of both mini-batches, moments and different regularizers like
2-norm and1-norm. Between iterations the step-size can be reduced based on either the negative log-likelihood or validation set performance. Contrasting available deep learning packages, our

Learning Deep Structured Models

11 pages including reference


2-norm and1-norm. Between iterations the step-size can be reduced based on either the negative log-likelihood or validation set performance. Contrasting available deep learning packages, our
functionFis speciﬁed using a general computation tree. Hence we support an arbitrarily nested
function structure composed of data, parameters and function prototypes (convolution, afﬁne function aka fully connected, dropout, local response normalization, pooling, rectiﬁed linear, sigmoid
and softmax units). The aforementioned library is accompanied by a program performing learning,
inference and gradient checks. To accommodate for large datasets it reads data from HDF5 storage
while a second thread simultaneously performs the computation. Google protocol buffers are employed to effectively specify the function Fwithout the need to modify any source code. We will
release this library upon publication. We believe that it will be useful for many researchers.
5
Accepted as a workshop contribution at ICLR 2015
Graph MLP Method H1= 128H1= 256H1= 512H1= 768H1= 1024
1st order Markov One LayerUnary only 8.60 / 61.32 10.80 / 64.41 12.50 / 65.69 12.95 / 66.66 13.40 / 67.02

Learning Deep Structured Models

11 pages including reference


Learning Deep Structured Models

11 pages including reference


PwTrain 39.95 / 81.14 48.25 / 84.45 52.65 / 86.24 57.10 / 87.61 62.90 / 89.49
PreTrainJoint 62.60 /88.03 65.80 /89.32 68.75 /90.47 68.60 /90.42 69.35 /90.75
Table 1: Word / Character accuracy. Performance improves as (1) joint-training is employed, (2) the
model is more structured, and (3) deeper unary classiﬁers are utilized. The number of hidden units
for the ﬁrst and second layer are denoted as H1andH2respectively.
abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
00.511.522.533.544.55
x 1043004005006007008009001000Neg. Log−Likelihood
Iteration
JointTrain
PwTrain
PreTrainJoint
Unary weights distance-1 edges distance-2 edges Neg. Log-Likelihood

Learning Deep Structured Models

11 pages including reference


Iteration
JointTrain
PwTrain
PreTrainJoint
Unary weights distance-1 edges distance-2 edges Neg. Log-Likelihood
Figure 4: (left) Subset of the learned unary weights. Pairwise weights (middle two panels), the
darker, the larger the weight. (right) Negative log-likelihood for different learning approaches.
3 E XPERIMENTAL EVALUATION
We demonstrate the performance of our model on two tasks: word recognition and image classiﬁcation. We investigate four strategies to learn the model parameters. ‘ Unary only ’ denotes training
only unary classiﬁers while ignoring the structure of the graphical model, i.e., pairwise weights are
equal to 0. ‘JointTrain ’ initializes all weights at random and trains them jointly. ‘ PwTrain ’ uses
piecewise training by ﬁrst training the unary potentials and then keeping them ﬁxed when learning the pairwise potentials. ‘ PreTrainJoint ’ pre-trains the unaries but jointly optimizes pairwise
weights as well as unary weights in a second step.
3.1 W ORD RECOGNITION : W ORD50
Our ﬁrst task consists of word recognition from noisy images. Towards this goal, we created a

Learning Deep Structured Models

11 pages including reference


3.1 W ORD RECOGNITION : W ORD50
Our ﬁrst task consists of word recognition from noisy images. Towards this goal, we created a
challenging dataset by randomly selecting 50 words, each consisting of ﬁve characters. We then
generated writing variations of each word as follows: we took the lower case characters from the
Chars74K dataset (de Campos et al., 2009), and inserted them in random background image patches
(similar to Larochelle et al. (2007)) by alpha matting, i.e., characters have transparency. To increase
the difﬁculty, we perturbed each character image of size 2828by scaling, rotation and translation.
As shown in Fig. 3 the task is very challenging, some characters are fairly difﬁcult to recognize even
for humans. We denote the resulting dataset ‘ Word50 .’ The training, validation and test sets have
10;000,2;000and2;000variations of words respectively.
We experimented with graphical models composed of unary and pairwise regions deﬁned over ﬁve

Learning Deep Structured Models

11 pages including reference


We experimented with graphical models composed of unary and pairwise regions deﬁned over ﬁve
random variables, one per character. We encode unary potentials fr(x;yi;wu)using multi-layer perceptrons (MLPs) with rectiﬁed linear units (ReLU). Unless otherwise stated, we deﬁne all pairwise
6
Accepted as a workshop contribution at ICLR 2015
H1=128 H1=256 H1=512 H1=768 H1=1024161820222426283032343638

Linear
PairH16
PairH32
PairH64
H2=32 H2=64 H2=128 H2=256 H2=512343638404244464850

Linear
PairH16
PairH32
PairH64
One-Layer MLP Chain Two-Layer MLP Chain
Figure 5: Learning non-linear pairwise functions: Word recognition as a function of the number
of hidden units for the unary potential. Colors represent different number of hidden units for the
pairwise potentials. The y-axis shows the word accuracy of using Linear function, or 16 (PairH16),
32 (PairH32), and 64 (PairH64) hidden units for the pairwise function.

Learning Deep Structured Models

11 pages including reference


32 (PairH32), and 64 (PairH64) hidden units for the pairwise function.
female/indoor/portrait sky/plant life/tree water/animals/sea animals/dog/indoor indoor/ﬂower/plant life
female/indoor/portrait sky/plant life/tree water/animals/sky animals/dog ;
Figure 6: Flickr test set images and some assigned tags as well as our predictions (bottom row).
interactions via
fr(x;yi;yj;wp) =X
mnWmn(yi=m;yj=n); (7)
wherer=fi;jg,wp=fWg,Wmnis the element of matrix W, andrefers to the indicator
function.
For all experiments, we share all unary weights across the nodes of the graphical model as well as all
pairwise weights for all edges. Note that due to the use of ReLU units, the negative log-likelihood
is non-smooth, non-linear and non-convex w.r.t. w. Because of the non-smoothness of F, we utilize
momentum based sub-gradient descent methods to estimate the weights. In particular, we use a

Learning Deep Structured Models

11 pages including reference


momentum based sub-gradient descent methods to estimate the weights. In particular, we use a
mini-batch size of 100, a step size of 0:01and a momentum of 0:95. If the unary potential is pretrained, the initial step size is reduced to 0:001. All the unary classiﬁers are trained with 100;000
iterations over mini-batches. For all experiments, the validation set is only used to decrease the
step size, i.e., if the accuracy on the validation set decreases, we reduce the step size by 0:5. We
use= 1, setcr= 1 for all regions r, and perform 10 message passing iterations to compute the
marginal beliefs b(x;y);rat step 2 in Fig. 2 when dealing with loopy models.
We experiment with two graphical models, Markov models of ﬁrst ( i.e., there are links only between
yiandyi+1) and second order ( i.e., there are links between yiandyi+1,yi+2) as well as two types of
unary potentials with varying degree of structure. We report two metrics, the average character and
word accuracy, which correspond to Hamming loss and zero-one loss respectively. Table 1 depicts
the results for the different models, learning strategies and number of hidden units. We observe the

Learning Deep Structured Models

11 pages including reference


word accuracy, which correspond to Hamming loss and zero-one loss respectively. Table 1 depicts
the results for the different models, learning strategies and number of hidden units. We observe the
following trends.
Joint training helps: Joint training with pre-trained unary classiﬁers (PreTrainJoint) outperforms
all the other approaches in almost all cases. The piecewise training method (PwTrain), unable to
adapt the non-linearities while learning pairwise weights, often leads to performance worse than
joint training.
Structure helps: Adding structure to the model is key to capture complex dependencies. As shown
in Table 1, more structured models ( i.e., second order Markov model) consistently improves performance.
7
Accepted as a workshop contribution at ICLR 2015
0.00 0.68 0.04 0.06 0.02 0.24 0.03 −0.00 −0.01 0.01 0.04 −0.00 −0.05 −0.01 0.07 −0.01 −0.00 −0.12 0.04 0.01 0.01 0.02 0.04 0.02

Learning Deep Structured Models

11 pages including reference


0.04 0.05 0.06 0.08 0.06 0.07 0.04 0.02 0.05 0.07 0.03 0.05 −0.07 0.04 0.06 −0.01 0.03 −0.01 0.07 −0.00 0.05 0.06 0.00 0.07
0.02 −0.03 −0.03 0.10 0.10 0.05 0.07 0.07 0.11 0.18 0.02 0.09 0.10 0.07 0.01 0.10 0.27 0.05 0.06 0.12 0.07 0.10 0.07 0.00female
people
indoor
baby
sea
portrait
transport
flower
sky
lake
structures
bird
plant life
food
male
clouds
water
animals
car
tree
dog
sunset
night
river
femalepeopleindoorbabyseaportraittransportflowerskylakestructuresbirdplant lifefoodmalecloudswateranimalscartreedogsunsetnightriver
0 5000 100000246810x 105
Time [s]Neg. Log−Likelihood

w/o blend
w blend
0 5000 100000200040006000800010000
Time [s]Training error

Learning Deep Structured Models

11 pages including reference


Time [s]Neg. Log−Likelihood

w/o blend
w blend
0 5000 100000200040006000800010000
Time [s]Training error

w/o blend
w blend
(a) (b)
Figure 7: (a) Correlation matrix ( i.e., pairwise potentials) learned on the Flickr dataset. (b) Blending
learning and inference speeds-up training signiﬁcantly.
Deep helps: We tested our models using one layer and two-layer perceptrons with both shortrange and long-range connections in the MRF. For the two-layer MLP, the number of hidden units
in the ﬁrst layer is ﬁxed to H1= 512 , and we varied the number of hidden units H2in the second
layer. As shown in Table 1 we observe that the deeper and the more structured the model, the better
the performance we achieve. As expected, performance also grows with the number of hidden units.
Efﬁciency: Using GPUs, it takes on average 0.064s per iteration for the 1st order Markov model
and 0.104s for the 2nd order Markov model. The time employed for training the one layer vs. the

Learning Deep Structured Models

11 pages including reference


and 0.104s for the 2nd order Markov model. The time employed for training the one layer vs. the
multi-layer models is approximately the same. Note that our approach is very efﬁcient, as this is the
time per iteration to train 831,166 weights.
Learned parameters: As shown in the left column of Fig. 4, the learned unary weights resemble
character strokes. The middle two panels show the learned pairwise weights for distance-1 edges
(i.e., edges with only neighboring connections) and distance-2 edges ( i.e., edges connecting every
other variable). For example, it shows that ‘q’ is likely to be followed by ‘u,’ and ‘e’ is likely to
be distance-2 away from ‘q’ in this dataset. On the right-most panel, we also show the negative
log-likelihood as a function of the number of joint training iterations. PreTrainJoint can achieve the
lowest cost value, while PwTrain has the highest value.
Non-linear pairwise functions: To further demonstrate the generality of our approach, we replaced
the linear pairwise function in Eq. (7) by a one-layer MLP, while keeping the other settings identical.

Learning Deep Structured Models

11 pages including reference


the linear pairwise function in Eq. (7) by a one-layer MLP, while keeping the other settings identical.
For this experiment we utilize a 1st order Markov model. As shown in Fig. 5, our model attains best
performance when using a non-linear pairwise function. We found 16 to 64 hidden units for the
non-linear pairwise function to be sufﬁcient for modeling the bi-gram combinations in this dataset.
In this case the largest model has 974,846 weights and training takes on average 0.068s per iteration.
3.2 I MAGE CLASSIFICATION : FLICKR
We next evaluate the importance of blending learning and inference. Towards this goal, we make
use of the Flickr dataset, which consists of 10;000training and 10;000test images from Flickr. The
task is to predict which of 38 possible tags should be assigned to each image. Fig. 6 shows some
examples. The graphical model has 38 binary random variables, each denoting the presence/absence
of a particular tag. We deﬁne the non-linear unaries fr(x;yi;wu)using the 8-layer deep-net architecture from Krizhevsky et al. (2013) followed by a 76-dimensional top layer. Hence the function

Learning Deep Structured Models

11 pages including reference


is composed out of two subsequent stacks of convolution, rectiﬁed linear (ReLU), pooling and local
response normalization units. Those are followed by three convolution–ReLU function pairs. Afterwards pooling is applied before two fully-connected–ReLU–dropout combinations are employed to
yield the input into a fully connected layer which ﬁnally computes the unary potentials. We employ
pairwise potentials similar to Eq. (7) which now fully model the correlations between any pair of
8
Accepted as a workshop contribution at ICLR 2015
output variables. This amounts to a total of 57;182;408parameters arising from the convolutional
units, fully connected units and corresponding biases as well as the pairwise weights.
We use a momentum based sub-gradient method for training with a mini-batch size of 300, a step
size of 0:0001 , a momentum of 0:95and set= 1 andcr= 18r. We initialized the deep-net
parameters using a model pre-trained on ImageNet (Deng et al., 2009). Our error metric is the
classiﬁcation error ( i.e., Hamming loss).
Joint training helps: The mean error for unary only potentials (‘Unary only’), piecewise training

Learning Deep Structured Models

11 pages including reference


Joint training helps: The mean error for unary only potentials (‘Unary only’), piecewise training
(‘PwTrain’) and joint pretraining (‘PreTrainJoint’) is 9.36%, 7.70% and 7.25% respectively. Similar
to the Word50 dataset we observe that joint training is beneﬁcial. We provide examples for perfect
(two left-most images), roughly accurate and failing predictions (right image) in Fig. 6.
Learned pairwise weights: In Fig. 7(a) we illustrate the learned correlations for a subset of the 38
classes. We observe that the class ‘people’ correlates highly with ‘female,’ ‘male,’ and ‘portrait.’
The ‘indoor’ tag does not co-occur with ‘sky,’ ‘structures,’ ‘plant life’ and ‘tree.’ ‘Sea’ appears
typically with ‘water,’ ‘clouds,’ ‘lake’ and ‘sky.’
Efﬁciency of Blending: To illustrate that blending is indeed beneﬁcial we compare the negative

Learning Deep Structured Models

11 pages including reference


Efﬁciency of Blending: To illustrate that blending is indeed beneﬁcial we compare the negative
log-likelihood and the training error as a function of run-time in Fig. 7(b). The standard approach
is limited to 20 iterations of message passing to avoid time-consuming, repeated computation of a
stopping criterion involving both the approximated log-partition function and its dual. As show in
Fig. 7(b) blending learning and inference speeds up parameter estimation signiﬁcantly. For larger
graphical models, we expect the differences to be even more signiﬁcant.
4 D ISCUSSION & C ONCLUSION
Joint training of neural networks and graphical models: Neural Networks have been incorporated as unary potentials in graphical models. One of the earliest works by Bridle (1990) jointly optimizes a system consisting of multilayer perceptrons and hidden Markov models for speech recognition. For document processing systems, Bottou et al. (1997) propose Graph Transformer Networks
to jointly optimize sub-tasks, such as word segmentation and character recognition. Several works
(Collobert et al., 2011; Peng et al., 2009; Ma et al., 2012; Do & Artieres, 2010; Prabhavalkar &

Learning Deep Structured Models

11 pages including reference


(Collobert et al., 2011; Peng et al., 2009; Ma et al., 2012; Do & Artieres, 2010; Prabhavalkar &
Fosler-Lussier, 2010; Morris & Fosler-Lussier, 2008) have extended the linear unary potential in
MRFs to incorporate non-linearities. However, they assume that exact inference can be performed
either via a forward-backward pass within the graphical model or dynamic programming. In contrast, in this paper we present learning algorithms for general graphical models, where inference is
hard. Moreover, all the previous works (except Do & Artieres (2010)) do not consider max-margin
loss during training which is incorporated into our framework by choosing = 0. More recently, Li
& Zemel (2014) use a hinge loss to learn the unary term deﬁned as a neural net, but keep the pairwise
potentials ﬁxed ( i.e., no joint training). Domke (2013) considers non-linear structured prediction and
decomposes the learning problem into a subset of logistic regressors, which require the parameter
updates to be run till convergence before updating the messages. Tompson et al. (2014) also jointly
train convolutional neural networks and a graphical model for pose estimation. However, the MRF

Learning Deep Structured Models

11 pages including reference


train convolutional neural networks and a graphical model for pose estimation. However, the MRF
inference procedure is approximated by their Spatial-Model which ignores the partition function.
Blending learning and inference: In this paper we deﬁned learning to be a min-max task. The
blending strategy, which was previously employed for learning log-linear models by (Meshi et al.,
2010; Hazan & Urtasun, 2010), amounts to converting the maximization task into a minimization
problem using its dual. Subsequently we make use of block-coordinate descent strategies to obtain
a more efﬁcient algorithm. Importantly any order of block-updates is possible. It remains an open
problem to ﬁnd the optimal tradeoff.
We have proposed an efﬁcient algorithm to learn deep models enriched to capture the dependencies
between the output variables. Our experiments on word prediction from noisy images and multiclass image classiﬁcation showed that the deeper and the more structured the model, the better the
performance we achieve. Furthermore, joint learning of all weights outperforms all other strategies. In the future we plan to learn deeper models in applications such as holistic semantic scene
understanding. We will also extend our approach to deal with hidden variables.
9
Accepted as a workshop contribution at ICLR 2015
ACKNOWLEDGMENTS

Learning Deep Structured Models

11 pages including reference


understanding. We will also extend our approach to deal with hidden variables.
9
Accepted as a workshop contribution at ICLR 2015
ACKNOWLEDGMENTS
We thank NVIDIA Corporation for the donation of GPUs used in this research. This work was
partially funded by ONR-N00014-14-1-0232.
REFERENCES
Bengio, Y ., Lamblin, P., Popovici, D., and Larochelle, H. Greedy Layer-Wise Training of Deep Networks. In
Proc. NIPS , 2007.
Bottou, L., Bengio, Y ., and LeCun, Y . Global training of document processing systems using graph transformer
networks. In Proc. CVPR , 1997.
Bridle, J. S. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. In Proc. NIPS , 1990.
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. Natural language processing
(almost) from scratch. JMLR , 2011.
de Campos, T. E., Babu, B. R., and Varma, M. Character recognition in natural images. In Proc. VISAPP , 2009.

Learning Deep Structured Models

11 pages including reference


de Campos, T. E., Babu, B. R., and Varma, M. Character recognition in natural images. In Proc. VISAPP , 2009.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image
database. In Proc. CVPR , 2009.
Deng, J., Ding, N., Jia, Y ., Frome, A., Murphy, K., Bengio, S., Li, Y ., Neven, H., and Adam, H. Large-Scale
Object Classiﬁcation using Label Relation Graphs. In Proc. ECCV , 2014.
Do, T.-M.-T. and Artieres, T. Neural conditional random ﬁelds. In Proc. AISTATS , 2010.
Domke, J. Structured Learning via Logistic Regression. In Proc. NIPS , 2013.
Eigen, D., Rolfe, J., Fergus, R., and LeCun, Y . Understanding Deep Architectures using a Recursive Convolutional Network. In Proc. ICLR , 2014.
Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and

Learning Deep Structured Models

11 pages including reference


Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and
semantic segmentation. In Proc. CVPR , 2014.
Hariharan, B., Arbel ´aez, P., Girshick, R., and Malik, J. Simultaneous detection and segmentation. In Proc.
ECCV , 2014.
Hazan, T. and Urtasun, R. A Primal-Dual Message-Passing Algorithm for Approximated Large Scale Structured Prediction. In Proc. NIPS , 2010.
Hinton, G. E. and Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science ,
2006.
Hinton, G. E., Sejnowski, T. J., and Ackley, D. H. Boltzmann Machines: Constraint Satisfaction Networks that
Learn. Technical report, University of Toronto, 1984.
Jia, Y . Caffe: An Open Source Convolutional Architecture for Fast Feature Embedding. http://caffe.
berkeleyvision.org/ , 2013.
Koller, D. and Friedman, N. Probabilistic Graphical Models: Principles and Techniques . MIT Press, 2009.

Learning Deep Structured Models

11 pages including reference