the MLP with two large hidden layers (several thousands) could reach nearly 0% training error,
but still did not manage to achieve good test error.
In the experiment results shown in Figure 14, we evaluate the impact of adding more training
data for the fully-connected MLP. As mentioned before for these experiments we have used a
MLP with three hidden layers where each layer has 2048 hidden units. The tanh( ) activation
function is used with 0.05 learning rate and minibatches of size 200.
As can be seen from the gure, adding more training examples did not help either training
or test error (both are near 50%, with training error slightly lower and test error slightly
higher), reinforcing the hypothesis that the dicult encountered is one of optimization, not of
regularization.
Figure 14: Training and test error bar charts for a regular MLP with 3 hidden layers. There
is no signicant improvement on the generalization error of the MLP as the new
training examples are introduced.
24
3.5 Experiments on Eect of Initializing with Hints
Initialization of the parameters in a neural network can have a big impact on the learning
and generalization (Glorot and Bengio, 2010). Previously Erhan et al. (2010) showed that


and generalization (Glorot and Bengio, 2010). Previously Erhan et al. (2010) showed that
initializing the parameters of a neural network with unsupervised pretraining guides the learning towards basins of attraction of local minima that provides better generalization from the
training dataset. In this section we analyze the eect of initializing the SMLP with hints and
then continuing without hints at the rest of the training. For experimental analysis of hints
based initialization, SMLP is trained for 1 training epoch using the hints and for 60 epochs it
is trained without hints on the 40k examples training set. We also compared the same architecture with the same hyperparameters, against to SMLP-nohints trained for 61 iterations on
the same dataset. After one iteration of hint-based training SMLP obtained 9% training error
and 39% test error. Following the hint based training, SMLP is trained without hints for 60
epochs, but at epoch 18, it already got 0% training and 0% test error. The hyperparameters
for this experiment and the experiment that the results shown for the SMLP-hints in Table 1
are the same. The test results for initialization with and without hints are shown on Figure 15.
This gure suggests that initializing with hints can give the same generalization performance
but training takes longer.

Knowledge Matters: Importance of Prior Information for Optimization

Summary:
We explore the effect of introducing prior information into the intermediate
level of neural networks for a learning task on which all the state-of-the-art
machine learning algorithms tested failed to learn. We motivate our work from
the hypothesis that humans learn such intermediate concepts from other
individuals via a form of supervision or guidance using a curriculum. The
experiments we have conducted provide positive evidence in favor of this
hypothesis. In our experiments, a two-tiered MLP architecture is trained on a
dataset with 64x64 binary inputs images, each image with three sprites. The
final task is to decide whether all the sprites are the same or one of them is
different. Sprites are pentomino tetris shapes and they are placed in an image
with different locations using scaling and rotation transformations. The first
part of the two-tiered MLP is pre-trained with intermediate-level targets being
the presence of sprites at each location, while the second part takes the
output of the first part as input and predicts the final task's target binary
event. The two-tiered MLP architecture, with a few tens of thousand examples,
was able to learn the task perfectly, whereas all other algorithms (include
unsupervised pre-training, but also traditional algorithms like SVMs, decision
trees and boosting) all perform no better than chance. We hypothesize that the
optimization difficulty involved when the intermediate pre-training is not
performed is due to the {\em composition} of two highly non-linear tasks. Our
findings are also consistent with hypotheses on cultural learning inspired by
the observations of optimization problems with deep learning, presumably
because of effective local minima.

37 Pages, 5 figures, 5 tables JMLR Special Topics on Representation
  Learning Submission


This gure suggests that initializing with hints can give the same generalization performance
but training takes longer.
Figure 15: Plots showing the test error of SMLP with random initialization vs initializing with
hint based training.
3.5.1 Further Experiments on Optimization for Pentomino Dataset
With extensive hyperparameter optimization and using standardization in the intermediate
level of the SMLP with softmax nonlinearity, SMLP-nohints was able to get 5.3% training and
25
6.7% test error on the 80k Pentomino training dataset. We used the 2050 hidden units for the
hidden layer of P1NN and 11 softmax output per patch. For the P2NN, we used 1024 hidden
units with sigmoid and learning rate 0.1 without using any adaptive learning rate method. This
SMLP uses a rectier nonlinearity for hidden layers of both P1NN and P2NN. Considering that
architecture uses softmax as the intermediate activation function of SMLP-nohints. It is very
likely that P1NN is trying to learn the presence of specic Pentomino shape in a given patch.
This architecture has a very large capacity in the P1NN, that probably provides it enough


This architecture has a very large capacity in the P1NN, that probably provides it enough
capacity to learn the presence of Pentomino shapes at each patch eortlessly.
An MLP with 2 hidden layers, each 1024 rectier units, was trained using LBFGS (the
implementation from the scipy.optimize library) on 40k training examples, with gradients computed on batches of 10000 examples at each iteration. However, after convergence of training,
the MLP was still doing chance on the test dataset.
We also observed that using linear units for the intermediate layer yields better generalization error without standardization compared to using activation functions such as sigmoid,
tanh and RELU for the intermediate layer. SMLP-nohints was able to get 25% generalization error with linear units without standardization whereas all the other activation functions
that has been tested failed to generalize with the same number of training iterations without
standardization and hints. This suggests that using non-linear intermediate-level activation
functions without standardization introduces an optimization diculty for the SMLP-nohints,
maybe because the intermediate level acts like a bottleneck in this architecture.
4. Conclusion and Discussion
In this paper we have shown an example of task which seems almost impossible to solve by
standard black-box machine learning algorithms, but can be almost perfectly solved when


4. Conclusion and Discussion
In this paper we have shown an example of task which seems almost impossible to solve by
standard black-box machine learning algorithms, but can be almost perfectly solved when
one encourages a semantics for the intermediate-level representation that is guided by prior
knowledge. The task has the particularity that it is dened by the composition of two nonlinear sub-tasks (object detection on one hand, and a non-linear logical operation similar to
XOR on the other hand).
What is interesting is that in the case of the neural network, we can compare two networks
with exactly the same architecture but a dierent pre-training, one of which uses the known
intermediate concepts to teach an intermediate representation to the network. With enough
capacity and training time they can overt but did not not capture the essence of the task, as
seen by test set performance.
We know that a structured deep network can learn the task, if it is initialized in the right
place, and do it from very few training examples. Furthermore we have shown that if one
pre-trains SMLP with hints for only one epoch, it can nail the task. But the exactly same
architecture which started training from random initialization, failed to generalize.
Consider the fact that even SMLP-nohints with standardization after being trained using


architecture which started training from random initialization, failed to generalize.
Consider the fact that even SMLP-nohints with standardization after being trained using
online SGD on 1046000 generated examples and still gets 27.5% test error. This is an indication
that the problem is not a regularization problem but possibly an inability to nd a good eective
local minima of generalization error .
What we hypothesize is that for most initializations and architectures (in particular the
fully-connected ones), although it is possible to nd a good eective local minimum of training
error when enough capacity is provided, it is dicult (without the proper initialization) to nd a
good local minimum of generalization error. On the other hand, when the network architecture
26
is constrained enough but still allows it to represent a good solution (such as the structured
MLP of our experiments), it seems that the optimization problem can still be dicult and even
training error remains stuck high if the standardization isn't used. Standardization obviously
makes the training objective of the SMLP easier to optimize and helps it to nd at least a
better eective local minimum of training error . This nding suggests that by using specic
architectural constraints and sometimes domain specic knowledge about the problem, one can


architectural constraints and sometimes domain specic knowledge about the problem, one can
alleviate the optimization diculty that generic neural network architectures face.
It could be that the combination of the network architecture and training procedure produces a training dynamics that tends to yield into these minima that are poor from the point of
view of generalization error, even when they manage to nail training error by providing enough
capacity. Of course, as the number of examples increases, we would expect this discrepancy to
decrease, but then the optimization problem could still make the task unfeasible in practice.
Note however that our preliminary experiments with increasing the training set size (8-fold) for
MLPs did not reveal signs of potential improvements in test error yet, as shown in Figure 14.
Even using online training on 545400 Pentomino examples, the SMLP-nohints architecture was
still doing far from perfect in terms of generalization error (Figure 12).
These ndings bring supporting evidence to the \Guided Learning Hypothesis" and \Deeper
Harder Hypothesis" from Bengio (2013a): higher level abstractions, which are expressed by
composing simpler concepts, are more dicult to learn (with the learner often getting in an
eective local minimum ), but that diculty can be overcome if another agent provides hints


eective local minimum ), but that diculty can be overcome if another agent provides hints
of the importance of learning other, intermediate-level abstractions which are relevant to the
task.
Many interesting questions remain open. Would a network without any guiding hint eventually nd the solution with a enough training time and/or with alternate parametrizations?
To what extent is ill-conditioning a core issue? The results with LBFGS were disappointing but
changes in the architectures (such as standardization of the intermediate level) seem to make
training much easier. Clearly, one can reach good solutions from an appropriate initialization,
pointing in the direction of an issue with local minima, but it may be that good solutions are
also reachable from other initializations, albeit going through a tortuous ill-conditioned path
in parameter space. Why did our attempts at learning the intermediate concepts in an unsupervised way fail? Are these results specic to the task we are testing or a limitation of the
unsupervised feature learning algorithm tested? Trying with many more unsupervised variants and exploring explanatory hypotheses for the observed failures could help us answer that.
Finally, and most ambitious, can we solve these kinds of problems if we allow a community
of learners to collaborate and collectively discover and combine partial solutions in order to
obtain solutions to more abstract tasks like the one presented here? Indeed, we would like to


of learners to collaborate and collectively discover and combine partial solutions in order to
obtain solutions to more abstract tasks like the one presented here? Indeed, we would like to
discover learning algorithms that can solve such tasks without the use of prior knowledge as
specic and strong as the one used in the SMLP here. These experiments could be inspired by
and inform us about potential mechanisms for collective learning through cultural evolutions
in human societies.
Acknowledgments
We would like to thank to the ICLR 2013 reviewers for their insightful comments, and NSERC,
CIFAR, Compute Canada and Canada Research Chairs for funding.
27
References
A. Ben-Hur and J. Weston. A user's guide to support vector machines. Methods in Molecular
Biology , 609:223{239, 2010.
Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep
networks. In NIPS'2006 , 2007.
Yoshua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine
Learning , 2(1):1{127, 2009. Also published as a book. Now Publishers, 2009.
Yoshua Bengio. Evolving culture vs local minima. In Growing Adaptive Machines: Integrating


Yoshua Bengio. Evolving culture vs local minima. In Growing Adaptive Machines: Integrating
Development and Learning in Articial Neural Networks , number also as ArXiv 1203.2990v1,
pages T. Kowaliw, N. Bredeche & R. Doursat, eds. Springer-Verlag, March 2013a. URL
http://arxiv.org/abs/1203.2990 .
Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures.
In K.-R. M uller, G. Montavon, and G. B. Orr, editors, Neural Networks: Tricks of the Trade .
Springer, 2013b.
Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning.
In L eon Bottou and Michael Littman, editors, Proceedings of the Twenty-sixth International
Conference on Machine Learning (ICML'09) . ACM, 2009a.
Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning.
InICML'09 , 2009b.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep


InICML'09 , 2009b.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep
learning: A review and new perspectives. Technical Report arXiv:1206.5538, U. Montreal,
2012. URL http://arxiv.org/abs/1206.5538 .
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep
learning: A review and new perspectives. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) , 2013.
James Bergstra, Olivier Breuleux, Fr ed eric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU
and GPU math expression compiler. In Proceedings of the Python for Scientic Computing
Conference (SciPy) , 2010.
L eon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of
COMPSTAT'2010 , pages 177{186. Springer, 2010.
Leo Breiman. Random forests. Machine Learning , 45(1):5{32, 2001.


COMPSTAT'2010 , pages 177{186. Springer, 2010.
Leo Breiman. Random forests. Machine Learning , 45(1):5{32, 2001.
D. C. Ciresan, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep big simple neural nets
for handwritten digit recognition. Neural Computation , 22:1{14, 2010.
Yann Dauphin and Yoshua Bengio. Big neural networks waste capacity. Technical Report
arXiv:1301.3583, Universite de Montreal, 2013.
Richard Dawkins. The Selsh Gene . Oxford University Press, 1976.
28
J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and
stochastic optimization. Journal of Machine Learning Research , 12:2121{2159, 2010.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent,
and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of
Machine Learning Research , 11:625{660, February 2010.
Fran cois Fleuret, Ting Li, Charles Dubout, Emma K Wampler, Steven Yantis, and Donald


Fran cois Fleuret, Ting Li, Charles Dubout, Emma K Wampler, Steven Yantis, and Donald
Geman. Comparing machines and humans on a visual categorization test. Proceedings of the
National Academy of Sciences , 108(43):17621{17625, 2011.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectier neural networks. In AISTATS ,
2011a.
Xavier Glorot and Yoshua Bengio. Understanding the diculty of training deep feedforward
neural networks. In JMLR W&CP: Proceedings of the Thirteenth International Conference
on Articial Intelligence and Statistics (AISTATS 2010) , volume 9, pages 249{256, May
2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectier neural networks. In
JMLR W&CP: Proceedings of the Fourteenth International Conference on Articial Intelligence and Statistics (AISTATS 2011) , April 2011b.
Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.
Maxout networks. In ICML , 2013.


Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.
Maxout networks. In ICML , 2013.
Maciej Henneberg. Decrease of human skull size in the holocene. Human biology , pages 395{405,
1988.
Maciej Henneberg and Maryna Steyn. Trends in cranial capacity and cranial index in subsaharan africa during the holocene. American journal of human biology , 5(4):473{479, 1993.
J. Henrich and R. McElreath. The evolution of cultural evolution. Evolutionary Anthropology:
Issues, News, and Reviews , 12(3):123{135, 2003.
Georey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep
belief nets. Neural Computation , 18:1527{1554, 2006.
Georey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. Technical
report, arXiv:1207.0580, 2012.


report, arXiv:1207.0580, 2012.
C.W. Hsu, C.C. Chang, C.J. Lin, et al. A practical guide to support vector classication, 2003.
Kevin Jarrett, Koray Kavukcuoglu, Marc'Aurelio Ranzato, and Yann LeCun. What is the
best multi-stage architecture for object recognition? In Proc. International Conference on
Computer Vision (ICCV'09) , pages 2146{2153. IEEE, 2009a.
Kevin Jarrett, Koray Kavukcuoglu, Marc'Aurelio Ranzato, and Yann LeCun. What is the best
multi-stage architecture for object recognition? In ICCV'09 , 2009b.
29
Faisal Khan, Xiaojin Zhu, and Bilge Mutlu. How do humans teach: On curriculum learning and
teaching dimension. In Advances in Neural Information Processing Systems 24 (NIPS'11) ,
pages 1449{1457, 2011.
Alex Krizhevsky, Ilya Sutskever, and Georey Hinton. ImageNet classication with deep
convolutional neural networks. In Advances in Neural Information Processing Systems 25
(NIPS'2012) . 2012.


convolutional neural networks. In Advances in Neural Information Processing Systems 25
(NIPS'2012) . 2012.
Kai A. Krueger and Peter Dayan. Flexible shaping: how learning in small steps helps. Cognition ,
110:380{394, 2009.
G. Kunapuli, K.P. Bennett, R. Maclin, and J.W. Shavlik. The adviceptron: Giving advice to
the perceptron. Proceedings of the Conference on Articial Neural Networks In Engineering
(ANNIE 2010) , 2010.
Hugo Larochelle, Yoshua Bengio, Jerome Louradour, and Pascal Lamblin. Exploring strategies
for training deep neural networks. Journal of Machine Learning Research , 10:1{40, 2009.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haner. Gradient-based learning applied to document
recognition. Proceedings of the IEEE , 86(11):2278{2324, 1998.
T.M. Mitchell. The need for biases in learning generalizations . Department of Computer
Science, Laboratory for Computer Science Research, Rutgers Univ., 1980.
T.M. Mitchell and S.B. Thrun. Explanation-based neural network learning for robot control.
Advances in Neural information processing systems , pages 287{287, 1993.


T.M. Mitchell and S.B. Thrun. Explanation-based neural network learning for robot control.
Advances in Neural information processing systems , pages 287{287, 1993.
R. Montague. Universal grammar. Theoria , 36(3):373{398, 1970.
V. Nair and G. E Hinton. Rectied linear units improve restricted Boltzmann machines. In
ICML'10 , 2010.
L.B.J.H.F.R.A. Olshen and C.J. Stone. Classication and regression trees. Belmont, Calif.:
Wadsworth , 1984.
Joseph O'Sullivan. Integrating initialization bias and search bias in neural network learning,
1996.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel,
P. Prettenhofer, R. Weiss, V. Dubourg, et al. Scikit-learn: Machine learning in python. The
Journal of Machine Learning Research , 12:2825{2830, 2011.
Gail B. Peterson. A day of great illumination: B. F. Skinner's discovery of shaping. Journal of
the Experimental Analysis of Behavior , 82(3):317{328, 2004.


the Experimental Analysis of Behavior , 82(3):317{328, 2004.
Tapani Raiko, Harri Valpola, and Yann LeCun. Deep learning made easier by linear transformations in perceptrons. In International Conference on Articial Intelligence and Statistics ,
pages 924{932, 2012.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive
auto-encoders: Explicit invariance during feature extraction. In ICML'2011 , 2011.
30
Salah Rifai, Yoshua Bengio, Yann Dauphin, and Pascal Vincent. A generative process for sampling contractive auto-encoders. In Proceedings of the Twenty-nine International Conference
on Machine Learning (ICML'12) . ACM, 2012. URL http://icml.cc/discuss/2012/590.
html .
R. Salakhutdinov and G.E. Hinton. Deep Boltzmann machines. In Proceedings of the Twelfth
International Conference on Articial Intelligence and Statistics (AISTATS 2009) , volume 8,
2009.
Burrhus F. Skinner. Reinforcement today. American Psychologist , 13:94{99, 1958.


2009.
Burrhus F. Skinner. Reinforcement today. American Psychologist , 13:94{99, 1958.
R.J. Solomono. A system for incremental learning based on algorithmic probability. In Proceedings of the Sixth Israeli Conference on Articial Intelligence, Computer Vision and Pattern
Recognition , pages 515{527. Citeseer, 1989.
G.G. Towell and J.W. Shavlik. Knowledge-based articial neural networks. Articial intelligence , 70(1):119{165, 1994.
Tommi Vatanen, Tapani Raiko, Harri Valpola, and Yann LeCun. Pushing stochastic gradient
towards second-order methods{backpropagation learning with transformations in nonlinearities. arXiv preprint arXiv:1301.3476 , 2013.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol.
Stacked denoising autoencoders: Learning useful representations in a deep network with a
local denoising criterion. Journal of Machine Learning Research , 11:3371{3408, December
2010.
Luis Von Ahn, Manuel Blum, Nicholas J Hopper, and John Langford. Captcha: Using hard


2010.
Luis Von Ahn, Manuel Blum, Nicholas J Hopper, and John Langford. Captcha: Using hard
ai problems for security. In Advances in CryptologyEUROCRYPT 2003 , pages 294{311.
Springer, 2003.
Jason Weston, Fr ed eric Ratle, and Ronan Collobert. Deep learning via semi-supervised embedding. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Proceedings of the Twenty-fth International Conference on Machine Learning (ICML'08) ,
pages 1168{1175, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-205-4. doi:
10.1145/1390156.1390303.
Matthew D Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint
arXiv:1212.5701 , 2012.
5. Appendix
5.1 Binary-Binary RBMs on Pentomino Dataset
We trained binary-binary RBMs (both visible and hidden are binary) on 8 8 patches extracted
from the Pentomino Dataset using PCD (stochastic maximum likelihood), a weight decay of
.0001 and a sparsity penalty9. We used 256 hidden units and trained by SGD with a batch


.0001 and a sparsity penalty9. We used 256 hidden units and trained by SGD with a batch
size of 32 and a annealing learning rate (Bengio, 2013b) starting from 1e-3 with annealing rate
9. implemented as TorontoSparsity in pylearn2, see the yaml le in the repository for more details
31
1.000015. The RBM is trained with momentum starting from 0.5. The biases are initialized to
-2 in order to get a sparse representation. The RBM is trained for 120 epochs (approximately
50 million updates).
After pretraining the RBM, its parameters are used to initialize the rst layer of an SMLPnohints network. As in the usual architecture of the SMLP-nohints on top of P1NN, there is
an intermediate layer. Both P1NN and the intermediate layer have a sigmoid nonlinearity, and
the intermediate layer has 11 units per location. This SMLP-nohints is trained with Adadelta
and standardization at the intermediate layer10.
0 5 10 15 20 25 30 35 40 45
Epoch0.00.10.20.30.40.5Error percentageRBM Test and Training Errors
Training Error
Test Error


0 5 10 15 20 25 30 35 40 45
Epoch0.00.10.20.30.40.5Error percentageRBM Test and Training Errors
Training Error
Test Error
Figure 16: Training and test errors of an SMLP-nohints network whose rst layer is pre-trained
as an RBM. Training error reduces to 0% at epoch 42, but test error is still chance.
5.2 Experimental Setup and Hyper-parameters
5.2.1 Decision Trees
We used the decision tree implementation in the scikit-learn (Pedregosa et al., 2011) python
package which is an implementation of the CART (Regression Trees) algorithm. The CART
algorithm constructs the decision tree recursively and partitions the input space such that the
samples belonging to the same category are grouped together (Olshen and Stone, 1984). We
used The Gini index as the impurity criteria. We evaluated the hyper-parameter congurations with a grid-search. We cross-validated the maximum depth ( maxdepth ) of the tree (for
preventing the algorithm to severely overt the training set) and minimum number of samples
10. In our auto-encoder experiments we directly fed features to P2NN without standardization and Adadelta.
32


10. In our auto-encoder experiments we directly fed features to P2NN without standardization and Adadelta.
32
Figure 17: Filters learned by the binary-binary RBM after training on the 40k examples. The
RBM did learn the edge structure of Pentomino shapes.
Figure 18: 100 samples generated from trained RBM. All the generated samples are valid Pentomino shapes.
33
required to create a split ( minsplit). 20 dierent congurations of hyper-parameter values were
evaluated. We obtained the best validation error with maxdepth = 300 andminsplit = 8.
5.2.2 Support Vector Machines
We used the \Support Vector Classier (SVC)" implementation from the scikit-learn package
which in turn uses the libsvm's Support Vector Machine (SVM) implementation. Kernelbased SVMs are non-parametric models that map the data into a high dimensional space and
separate dierent classes with hyperplane(s) such that the support vectors for each category
will be separated by a large margin. We cross-validated three hyper-parameters of the model
using grid-search: C,
and the type of kernel( kerneltype).Cis the penalty term (weight


using grid-search: C,
and the type of kernel( kerneltype).Cis the penalty term (weight
decay) for the SVM and
is a hyper-parameter that controls the width of the Gaussian for
the RBF kernel. For the polynomial kernel,
controls the
exibility of the classier (degree
of the polynomial) as the number of parameters increases (Hsu et al., 2003; Ben-Hur and
Weston, 2010). We evaluated forty-two hyper-parameter congurations. That includes, two
kernel types:fRBF; Polynomial g; three gammas:f1e2;1e3;1e4gfor the RBF kernel,
f1;2;5gfor the polynomial kernel, and seven Cvalues among:f0:1;1;2;4;8;10;16g. As a
result of the grid search and cross-validation, we have obtained the best test error by using the
RBF kernel, with C= 2 and
= 1.
5.2.3 Multi Layer Perceptron
We have our own implementation of Multi Layer Perceptron based on the Theano (Bergstra
et al., 2010) machine learning libraries. We have selected 2 hidden layers, the rectier activation


et al., 2010) machine learning libraries. We have selected 2 hidden layers, the rectier activation
function, and 2048 hidden units per layer. We cross-validated three hyper-parameters of the
model using random-search, sampling the learning rates in log-domain, and selecting L1
andL2 regularization penalty coecients in sets of xed values, evaluating 64 hyperparameter
values. The range of the hyperparameter values are 2[0:0001;1],L12f0:;1e6;1e5;1e4g
andL22f0;1e6;1e5g. As a result, the following were selected: L1 = 1e6,L2 = 1e5
and= 0:05.
5.2.4 Random Forests
We used scikit-learn's implementation of \Random Forests" decision tree learning. The Random Forests algorithm creates an ensemble of decision trees by randomly selecting for each tree
a subset of features and applying bagging to combine the individual decision trees (Breiman,
2001). We have used grid-search and cross-validated the maxdepth ,minsplit, and number


2001). We have used grid-search and cross-validated the maxdepth ,minsplit, and number
of trees (nestimators ). We have done the grid-search on the following hyperparameter values,nestimators2 f5;10;15;25;50g,maxdepth2 f100;300;600;900g, andminsplits2
f1;4;16g. We obtained the best validation error with maxdepth = 300,minsplit = 4 and
nestimators = 10.
5.2.5 k-Nearest Neighbors
We used scikit-learn's implementation of k-Nearest Neighbors (k-NN). k-NN is an instancebased, lazy learning algorithm that selects the training examples closest in Euclidean distance
to the input query. It assigns a class label to the test example based on the categories of the
kclosest neighbors. The hyper-parameters we have evaluated in the cross-validation are the
number of neighbors ( k) andweights . Theweights hyper-parameter can be either \uniform" or
34
\distance". With \uniform", the value assigned to the query point is computed by the majority
vote of the nearest neighbors. With \distance", each value assigned to the query point is
computed by weighted majority votes where the weights are computed with the inverse distance


vote of the nearest neighbors. With \distance", each value assigned to the query point is
computed by weighted majority votes where the weights are computed with the inverse distance
between the query point and the neighbors. We have used nneighbours2f1;2;4;6;8;12gand
weights2f"uniform ";"distance "gfor hyper-parameter search. As a result of cross-validation
and grid search, we obtained the best validation error with k= 2 andweights =\uniform".
5.2.6 Convolutional Neural Nets
We used a Theano (Bergstra et al., 2010) implementation of Convolutional Neural Networks
(CNN) from the deep learning tutorial at deeplearning.net , which is based on a vanilla
version of a CNN LeCun et al. (1998). Our CNN has two convolutional layers. Following
each convolutional layer, we have a max-pooling layer. On top of the convolution-poolingconvolution-pooling layers there is an MLP with one hidden layer. In the cross-validation we
have sampled 36 learning rates in log-domain in the range [0 :0001;1] and the number of lters


have sampled 36 learning rates in log-domain in the range [0 :0001;1] and the number of lters
from the range [10 ;20;30;40;50;60] uniformly. For the rst convolutional layer we used 9 9
receptive elds in order to guarantee that each object ts inside the receptive eld. As a result
of random hyperparameter search and doing manual hyperparameter search on the validation
dataset, the following values were selected:
The number of features used for the rst layer is 30 and the second layer is 60.
For the second convolutional layer, 7 7 receptive elds. The stride for both convolutional
layers is 1.
Convolved images are downsampled by a factor of 2 2 at each pooling operation.
The learning rate for CNN is 0.01 and it was trained for 8 epochs.
5.2.7 Maxout Convolutional Neural Nets
We used the pylearn2 ( https://github.com/lisa-lab/pylearn2 ) implementation of maxout
convolutional networks (Goodfellow et al., 2013). There are two convolutional layers in the


convolutional networks (Goodfellow et al., 2013). There are two convolutional layers in the
selected architecture, without any pooling. In the last convolutional layer, there is a maxout
non-linearity. The following were selected by cross-validation: learning rate, number of channels
for the both convolution layers, number of kernels for the second layer and number of units and
pieces per maxout unit in the last layer, a linearly decaying learning rate, momentum starting
from 0.5 and saturating to 0.8 at the 200'th epoch. Random search for the hyperparameters
was used to evaluate 48 dierent hyperparameter congurations on the validation dataset. For
the rst convolutional layer, 8 8 kernels were selected to make sure that each Pentomino shape
ts into the kernel. Early stopping was used and test error on the model that has the best
validation error is reported. Using norm constraint on the fan-in of the nal softmax units
yields slightly better result on the validation dataset.
As a result of cross-validation and manually tuning the hyperparameters we used the following hyperparameters:
16 channels per convolutional layer. 600 hidden units for the maxout layer.
6x6 kernels for the second convolutional layer.
35


16 channels per convolutional layer. 600 hidden units for the maxout layer.
6x6 kernels for the second convolutional layer.
35
5 pieces for the convolution layers and 4 pieces for the maxout layer per maxout units.
We decayed the learning rate by the factor of 0.001 and the initial learning rate is 0.026367.
But we scaled the learning rate of the second convolutional layer by a constant factor of
0.6.
The norm constraint (on the incoming weights of each unit) is 1.9365.
Figure 19 shows the rst layer lters of the maxout convolutional net, after being trained
on the 80k training set for 85 epochs.
Figure 19: Maxout convolutional net rst layer lters. Most of the lters were able to learn
the basic edge structure of the Pentomino shapes.
5.2.8 Stacked Denoising Auto-Encoders
Denoising Auto-Encoders (DAE) are a form of regularized auto-encoder (Bengio et al., 2013).
The DAE forces the hidden layer to discover more robust features and prevents it from simply
learning the identity by reconstructing the input from a corrupted version of it (Vincent et al.,


The DAE forces the hidden layer to discover more robust features and prevents it from simply
learning the identity by reconstructing the input from a corrupted version of it (Vincent et al.,
2010). Two DAEs were stacked, resulting in an unsupervised transformation with two hidden
layers of 1024 units each. Parameters of all layers are then ne-tuned with supervised netuning using logistic regression as the classier and SGD as the gradient-based optimization
algorithm. The stochastic corruption process is binomial (0 or 1 replacing each input value,
with probability 0.2). The selected learning rate is 0= 0:01 for the DAe and 1= 0:1 for
supervised ne-tuning. Both L1 and L2 penalty for the DAEs and for the logistic regression
layer are set to 1e-6.
CAE+MLP with Supervised Finetuning: A regularized auto-encoder which sometimes
outperforms the DAE is the Contractive Auto-Encoder (CAE), (Rifai et al., 2012), which
penalizes the Frobenius norm of the Jacobian matrix of derivatives of the hidden units with
respect to the CAE's inputs. The CAE serves as pre-training for an MLP, and in the supervised


respect to the CAE's inputs. The CAE serves as pre-training for an MLP, and in the supervised
ne-tuning state, the Adagrad method was used to automatically tune the learning rate (Duchi
et al., 2010).
After training a CAE with 100 sigmoidal units patch-wise, the features extracted on each
patch are concatenated and fed as input to an MLP. The selected Jacobian penalty coecient
is 2, the learning rate for pre-training is 0.082 with batch size of 200 and 200 epochs of unsupervised learning are performed on the training set. For supervised netuning, the learning
rate is 0.12 over 100 epochs, L1 and L2 regularization penalty terms respectively are 1e-4 and
1e-6, and the top-level MLP has 6400 hidden units.
36
Greedy Layerwise CAE+DAE Supervised Finetuning: For this experiment we stack a
CAE with sigmoid non-linearities and then a DAE with rectier non-linearities during the pretraining phase. As recommended by Glorot et al. (2011b) we have used a softplus nonlinearity
for reconstruction, softplus (x) =log(1 +ex). We used an L1 penalty on the rectier outputs


for reconstruction, softplus (x) =log(1 +ex). We used an L1 penalty on the rectier outputs
to obtain a sparser representation with rectier non-linearity and L2 regularization to keep the
non-zero weights small.
The main dierence between the DAE and CAE is that the DAE yields more robust reconstruction whereas the CAE obtains more robust features (Rifai et al., 2011).
As seen on Figure 7 the weights U and V are shared on each patch and we concatenate the
outputs of the last auto-encoder on each patch to feed it as an input to an MLP with a large
hidden layer.
We used 400 hidden units for the CAE and 100 hidden units for DAE. The learning rate
used for the CAE is 0.82 and for DAE it is 9*1e-3. The corruption level for the DAE (binomial
noise) is 0.25 and the contraction level for the CAE is 2.0. The L1 regularization penalty for
the DAE is 2.25*1e-4 and the L2 penalty is 9.5*1e-5. For the supervised netuning phase the


the DAE is 2.25*1e-4 and the L2 penalty is 9.5*1e-5. For the supervised netuning phase the
learning rate used is 4*1e-4 with L1 and L2 penalties respectively 1e-5 and 1e-6. The top-level
MLP has 6400 hidden units. The auto-encoders are each trained for 150 epochs while the whole
MLP is ne-tuned for 50 epochs.
Greedy Layerwise DAE+DAE Supervised Finetuning: For this architecture, we have
trained two layers of denoising auto-encoders greedily and performed supervised netuning
after unsupervised pre-training. The motivation for using two denoising auto-encoders is the
fact that rectier nonlinearities work well with the deep networks but it is dicult to train
CAEs with the rectier non-linearity. We have used the same type of denoising auto-encoder
that is used for the greedy layerwise CAE+DAE supervised netuning experiment.
In this experiment we have used 400 hidden units for the rst layer DAE and 100 hidden
units for the second layer DAE. The other hyperparameters for DAE and supervised netuning


units for the second layer DAE. The other hyperparameters for DAE and supervised netuning
are the same as with the CAE+DAE MLP Supervised Finetuning experiment.
37


DeepPose: Human Pose Estimation via Deep Neural Networks
Alexander Toshev Christian Szegedy
Google
1600 Amphitheatre Pkwy
Mountain View, CA 94043
toshev,szegedy@google.com
Figure 1. Besides extreme variability in articulations, many of the
joints are barely visible. We can guess the location of the right
arm in the left image only because we see the rest of the pose and
anticipate the motion or activity of the person. Similarly, the left
body half of the person on the right is not visible at all. These
are examples of the need for holistic reasoning . We believe that
DNNs can naturally provide such type of reasoning.
Abstract
We propose a method for human pose estimation based
on Deep Neural Networks (DNNs). The pose estimation
is formulated as a DNN-based regression problem towards
body joints. We present a cascade of such DNN regressors which results in high precision pose estimates. The
approach has the advantage of reasoning about pose in a
holistic fashion and has a simple but yet powerful formulation which capitalizes on recent advances in Deep Learning. We present a detailed empirical analysis with state-ofart or better performance on four academic benchmarks of
diverse real-world images.
1. Introduction
The problem of human pose estimation, deﬁned as the

Summary:
We propose a method for human pose estimation based on Deep Neural Networks
(DNNs). The pose estimation is formulated as a DNN-based regression problem
towards body joints. We present a cascade of such DNN regressors which results
in high precision pose estimates. The approach has the advantage of reasoning
about pose in a holistic fashion and has a simple but yet powerful formulation
which capitalizes on recent advances in Deep Learning. We present a detailed
empirical analysis with state-of-art or better performance on four academic
benchmarks of diverse real-world images.

IEEE Conference on Computer Vision and Pattern Recognition, 2014


diverse real-world images.
1. Introduction
The problem of human pose estimation, deﬁned as the
problem of localization of human joints, has enjoyed substantial attention in the computer vision community. In
Fig. 1, one can see some of the challenges of this problem – strong articulations, small and barely visible joints,
occlusions and the need to capture the context.
The main stream of work in this ﬁeld has been motivatedmainly by the ﬁrst challenge, the need to search in the large
space of all possible articulated poses. Part-based models
lend themselves naturally to model articulations ([16, 8])
and in the recent years a variety of models with efﬁcient
inference have been proposed ([6, 19]).
The above efﬁciency, however, is achieved at the cost of
limited expressiveness – the use of local detectors, which
reason in many cases about a single part, and most importantly by modeling only a small subset of all interactions
between body parts. These limitations, as exempliﬁed in
Fig. 1, have been recognized and methods reasoning about
pose in a holistic manner have been proposed [15, 21] but
with limited success in real-world problems.
In this work we ascribe to this holistic view of human
pose estimation. We capitalize on recent developments of


with limited success in real-world problems.
In this work we ascribe to this holistic view of human
pose estimation. We capitalize on recent developments of
deep learning and propose a novel algorithm based on a
Deep Neural Network (DNN). DNNs have shown outstanding performance on visual classiﬁcation tasks [14] and more
recently on object localization [23, 9]. However, the question of applying DNNs for precise localization of articulated
objects has largely remained unanswered. In this paper we
attempt to cast a light on this question and present a simple
and yet powerful formulation of holistic human pose estimation as a DNN.
We formulate the pose estimation as a joint regression
problem and show how to successfully cast it in DNN settings. The location of each body joint is regressed to using
as an input the full image and a 7-layered generic convolutional DNN. There are two advantages of this formulation.
First, the DNN is capable of capturing the full context of
each body joint – each joint regressor uses the full image
as a signal. Second, the approach is substantially simpler
to formulate than methods based on graphical models – no
need to explicitly design feature representations and detectors for parts; no need to explicitly design a model topology
and interactions between joints. Instead, we show that a
generic convolutional DNN can be learned for this problem.


and interactions between joints. Instead, we show that a
generic convolutional DNN can be learned for this problem.
Further, we propose a cascade of DNN-based pose predictors. Such a cascade allows for increased precision of
1arXiv:1312.4659v3  [cs.CV]  20 Aug 2014
joint localization. Starting with an initial pose estimation,
based on the full image, we learn DNN-based regressors
which reﬁnes the joint predictions by using higher resolution sub-images.
We show state-of-art results or better than state-of-art on
four widely used benchmarks against all reported results.
We show that our approach performs well on images of people which exhibit strong variation in appearance as well as
articulations. Finally, we show generalization performance
by cross-dataset evaluation.
2. Related Work
The idea of representing articulated objects in general,
and human pose in particular, as a graph of parts has been
advocated from the early days of computer vision [16]. The
so called Pictorial Strictures (PSs), introduced by Fishler
and Elschlager [8], were made tractable and practical by
Felzenszwalb and Huttenlocher [6] using the distance transform trick. As a result, a wide variety of PS-based models


Felzenszwalb and Huttenlocher [6] using the distance transform trick. As a result, a wide variety of PS-based models
with practical signiﬁcance were subsequently developed.
The above tractability, however, comes with the limitation of having a tree-based pose models with simple binary
potential not depending on image data. As a result, research
has focused on enriching the representational power of the
models while maintaining tractability. Earlier attempts to
achieve this were based on richer part detectors [19, 1, 4].
More recently, a wide variety of models expressing complex
joint relationships were proposed. Yang and Ramanan [27]
use a mixture model of parts. Mixture models on the full
model scale, by having mixture of PSs, have been studied
by Johnson and Everingham [13]. Richer higher-order spatial relationships were captured in a hierarchical model by
Tian et al. [25]. A different approach to capture higherorder relationship is through image-dependent PS models,
which can be estimated via a global classiﬁer [26, 20, 18].
Approaches which ascribe to our philosophy of reasoning about pose in a holistic manner have shown limited
practicality. Mori and Malik [15] try to ﬁnd for each test
image the closest exemplar from a set of labeled images


practicality. Mori and Malik [15] try to ﬁnd for each test
image the closest exemplar from a set of labeled images
and transfer the joint locations. A similar nearest neighbor
setup is employed by Shakhnarovich et al. [21], who however use locality sensitive hashing. More recently, Gkioxari
et al. [10] propose a semi-global classiﬁer for part conﬁguration. This formulation has shown very good results on
real-world data, however, it is based on linear classiﬁers
with less expressive representation than ours and is tested
on arms only. Finally, the idea of pose regression has been
employed by Ionescu et al. [11], however they reason about
3D pose.
The closest work to ours uses convolution NNs together
with Neighborhood Component Analysis to regress toward
a point in an embedding representing pose [24]. However,
this work does not employ a cascade of networks. Cascadesof DNN regressors have been used for localization, however
of facial points [22]. On the related problem of face pose
estimation, Osadchy et al. [17] employ a NN-based pose
embedding trained with a contrastive loss.
3. Deep Learning Model for Pose Estimation


estimation, Osadchy et al. [17] employ a NN-based pose
embedding trained with a contrastive loss.
3. Deep Learning Model for Pose Estimation
We use the following notation. To express a pose, we encode the locations of all kbody joints in pose vector deﬁned
asy= (:::;yT
i;:::)T;i2f1;:::;kg, where yicontains
thexandycoordinates of the ithjoint. A labeled image is
denoted by (x;y)wherexstands for the image data and y
is the ground truth pose vector.
Further, since the joint coordinates are in absolute image
coordinates, it proves beneﬁcial to normalize them w. r. t. a
boxbbounding the human body or parts of it. In a trivial
case, the box can denote the full image. Such a box is deﬁned by its center bc2R2as well as width bwand height
bh:b= (bc;bw;bh). Then the joint yican be translated by
the box center and scaled by the box size which we refer to
as normalization by b:
N(yi;b) =1=bw 0
0 1=bh
(yibc) (1)


as normalization by b:
N(yi;b) =1=bw 0
0 1=bh
(yibc) (1)
Further, we can apply the same normalization to the elements of pose vector N(y;b) = (:::;N (yi;b)T;:::)Tresulting in a normalized pose vector . Finally, with a slight
abuse of notation, we use N(x;b)to denote a crop of the
imagexby the bounding box b, which de facto normalizes
the image by the box. For brevity we denote by N()normalization with bbeing the full image box.
3.1. Pose Estimation as DNN-based Regression
In this work, we treat the problem of pose estimation as
regression, where the we train and use a function  (x;)2
R2kwhich for an image xregresses to a normalized pose
vector, where denotes the parameters of the model. Thus,
using the normalization transformation from Eq. (1) the
pose prediction yin absolute image coordinates reads
y=N1( (N(x);)) (2)
Despite its simple formulation, the power and complexity of the method is in  , which is based on a convolutional


Despite its simple formulation, the power and complexity of the method is in  , which is based on a convolutional
Deep Neural Network (DNN). Such a convolutional network consists of several layers – each being a linear transformation followed by a non-linear one. The ﬁrst layer takes
as input an image of predeﬁned size and has a size equal to
the number of pixels times three color channels. The last
layer outputs the target values of the regression, in our case
2kjoint coordinates.
We base the architecture of the  on the work by
Krizhevsky et al. [14] for image classiﬁcation since it has
shown outstanding results on object localization as well
(xi, yi)
(x(s-1)i, y (s-1) i)xsi - x(s-1)iysi - y(s-1)iInitial stageStage s
send reﬁned values to next stage220 x 220
DNN-based regressor
27 x 27 x 128
13 x 13 x 192
13 x 13 x192
13 x 13 x192
4096
4096
55 x 55 x 48xiyi...DNN-based reﬁner
27 x 27 x 128
13 x 13 x 192
13 x 13 x192


4096
55 x 55 x 48xiyi...DNN-based reﬁner
27 x 27 x 128
13 x 13 x 192
13 x 13 x192
13 x 13 x192
4096
4096
55 x 55 x 48Figure 2. Left: schematic view of the DNN-based pose regression. We visualize the network layers with their corresponding dimensions,
where convolutional layers are in blue, while fully connected ones are in green. We do not show the parameter free layers. Right: at stage
s, a reﬁning regressor is applied on a sub image to reﬁne a prediction from the previous stage.
[23]. In a nutshell, the network consists of 7layers (see
Fig. 2 left). Denote by Ca convolutional layer, by LRN
a local response normalization layer, Pa pooling layer
and byFa fully connected layer. Only CandFlayers
contain learnable parameters, while the rest are parameter free. Both CandFlayers consist of a linear transformation followed by a nonlinear one, which in our case
is a rectiﬁed linear unit. For Clayers, the size is deﬁned as widthheightdepth, where the ﬁrst two dimensions have a spatial meaning while the depth deﬁnes


the number of ﬁlters. If we write the size of each layer in
parentheses, then the network can be described concisely
asC(555596)LRNPC(2727256)
LRNPC(1313384)C(1313384)
C(1313256)PF(4096)F(4096) . The ﬁlter
size for the ﬁrst two Clayers is 1111and55and for
the remaining three is 33. Pooling is applied after three
layers and contributes to increased performance despite the
reduction of resolution. The input to the net is an image
of220220which via stride of 4is fed into the network.
The total number of parameters in the above model is about
40M. For further details, we refer the reader to [14].
The use of a generic DNN architecture is motivated by
its outstanding results on both classiﬁcation and localization
problems. In the experimental section we show that such a
generic architecture can be used to learn a model resulting
in state-of-art or better performance on pose estimation as
well. Further, such a model is a truly holistic one — the


generic architecture can be used to learn a model resulting
in state-of-art or better performance on pose estimation as
well. Further, such a model is a truly holistic one — the
ﬁnal joint location estimate is based on a complex nonlinear
transformation of the full image.
Additionally, the use of a DNN obviates the need to design a domain speciﬁc pose model. Instead such a model
and the features are learned from the data. Although the regression loss does not model explicit interactions between
joints, such are implicitly captured by all of the 7hidden
layers – all the internal features are shared by all joint regressors.
Training The difference to [14] is the loss. Instead of a
classiﬁcation loss, we train a linear regression on top of thelast network layer to predict a pose vector by minimizing
L2distance between the prediction and the true pose vector. Since the ground truth pose vector is deﬁned in absolute image coordinates and poses vary in size from image to
image, we normalize our training set Dusing the normalization from Eq. (1):
DN=f(N(x);N(y))j(x;y)2Dg (3)
Then theL2loss for obtaining optimal network parameters
reads:
arg min
X


Then theL2loss for obtaining optimal network parameters
reads:
arg min
X
(x;y)2DNkX
i=1jjyi i(x;)jj2
2 (4)
For clarity we write out the optimization over individual
joints. It should be noted, that the above objective can
be used even if for some images not all joints are labeled.
In this case, the corresponding terms in the sum would be
omitted.
The above parameters are optimized for using Backpropagation in a distributed online implementation. For
each mini-batch of size 128, adaptive gradient updates are
computed [3]. The learning rate, as the most important parameter, is set to 0:0005 . Since the model has large number
of parameters and the used datasets are of relatively small
size, we augment the data using large number of randomly
translated image crops (see Sec. 3.2), left/right ﬂips as well
as DropOut regularization for the Flayers set to 0:6.
3.2. Cascade of Pose Regressors
The pose formulation from the previous section has the
advantage that the joint estimation is based on the full image and thus relies on context. However, due to its ﬁxed
input size of 220220, the network has limited capacity


input size of 220220, the network has limited capacity
to look at detail – it learns ﬁlters capturing pose properties
at coarse scale. These are necessary to estimate rough pose
but insufﬁcient to always precisely localize the body joints.
Note that we cannot easily increase the input size since this
will increase the already large number of parameters. In order to achieve better precision, we propose to train a cascade
of pose regressors. At the ﬁrst stage, the cascade starts off
by estimating an initial pose as outlined in the previous section. At subsequent stages, additional DNN regressors are
trained to predict a displacement of the joint locations from
previous stage to the true location. Thus, each subsequent
stage can be thought of as a reﬁnement of the currently predicted pose, as shown in Fig. 2.
Further, each subsequent stage uses the predicted joint
locations to focus on the relevant parts of the image – subimages are cropped around the predicted joint location from
previous stage and the pose displacement regressor for this
joint is applied on this sub-image. In this way, subsequent
pose regressors see higher resolution images and thus learn
features for ﬁner scales which ultimately leads to higher
precision.
We use the same network architecture for all stages of
the cascade but learn different network parameters. For


features for ﬁner scales which ultimately leads to higher
precision.
We use the same network architecture for all stages of
the cascade but learn different network parameters. For
stages2 f1;:::;Sgof totalScascade stages, we denote bysthe learned network parameters. Thus, the
pose displacement regressor reads  (x;s). To reﬁne a
given joint location yiwe will consider a joint bounding
boxbicapturing the sub-image around yi:bi(y;) =
(yi;diam(y);diam(y))having as center the i-th joint
and as dimension the pose diameter scaled by . The diameter diam (y)of the pose is deﬁned as the distance between
opposing joints on the human torso, such as left shoulder
and right hip, and depends on the concrete pose deﬁnition
and dataset.
Using the above notation, at the stage s= 1we start with
a bounding box b0which either encloses the full image or
is obtained by a person detector. We obtain an initial pose:
Stage 1 : y1 N1( (N(x;b0);1);b0)(5)


is obtained by a person detector. We obtain an initial pose:
Stage 1 : y1 N1( (N(x;b0);1);b0)(5)
At each subsequent stage s2, for all joints i2f1;:::;kg
we regress ﬁrst towards a reﬁnement displacement ys
i
y(s1)
i by applying a regressor on the sub image deﬁned
byb(s1)
i from previous stage (s1). Then, we estimate
new joint boxes bs
i:
Stages:ys
i y(s1)
i +N1( i(N(x;b);s);b)(6)
forb=b(s1)
i
bs
i (ys
i;diam(ys);diam(ys)) (7)
We apply the cascade for a ﬁxed number of stages S,
which is determined as explained in Sec. 4.1.
Training The network parameters 1are trained as
outlined in Sec. 3.1, Eq. (4). At subsequent stages


Training The network parameters 1are trained as
outlined in Sec. 3.1, Eq. (4). At subsequent stages
s2, the training is done identically with one important difference. Each joint ifrom a training example(x;y)is normalized using a different bounding box(y(s1)
i;diam(y(s1));diam(y(s1)))– the one centered at the prediction for the same joint obtained from previous stage – so that we condition the training of the stage
based on the model from previous stage.
Since deep learning methods have large capacity, we
augment the training data by using multiple normalizations
for each image and joint. Instead of using the prediction
from previous stage only, we generate simulated predictions. This is done by randomly displacing the ground truth
location for joint iby a vector sampled at random from a
2-dimensional Normal distribution N(s1)
i with mean and
variance equal to the mean and variance of the observed displacements (y(s1)
iyi)across all examples in the training data. The full augmented training data can be deﬁned
by ﬁrst sampling an example and a joint from the original
data at uniform and then generating a simulated prediction


by ﬁrst sampling an example and a joint from the original
data at uniform and then generating a simulated prediction
based on a sampled displacement fromN(s1)
i :
Ds
A=f(N(x;b);N(yi;b))j
(x;yi)D;N(s1)
i;
b= (yi+;diam(y))g
The training objective for cascade stage sis done as in
Eq. (4) by taking extra care to use the correct normalization
for each joint:
s= arg min
X
(x;yi)2Ds
Ajjyi i(x;)jj2
2 (8)
4. Empirical Evaluation
4.1. Setup
Datasets There is a wide variety of benchmarks for human pose estimation. In this work we use datasets, which
have large number of training examples sufﬁcient to train a
large model such as the proposed DNN, as well as are realistic and challenging.
The ﬁrst dataset we use is Frames Labeled In Cinema
(FLIC), introduced by [20], which consists of 4000 training and 1000 test images obtained from popular Hollywood
movies. The images contain people in diverse poses and especially diverse clothing. For each labeled human, 10upper


movies. The images contain people in diverse poses and especially diverse clothing. For each labeled human, 10upper
body joints are labeled.
The second dataset we use is Leeds Sports Dataset [12]
and its extension [13], which we will jointly denote by LSP.
Combined they contain 11000 training and 1000 testing images. These are images from sports activities and as such
are quite challenging in terms of appearance and especially
articulations. In addition, the majority of people have 150
pixel height which makes the pose estimation even more
challenging. In this dataset, for each person the full body is
labeled with total 14joints.
For all of the above datasets, we deﬁne the diameter of a
poseyto be the distance between a shoulder and hip from
opposing sides and denote it by diam (y). It should be noted,
that the joints in all datasets are arranged in a tree kinematically mimicking the human body. This allows for a deﬁnition of a limb being a pair of neighboring joints in the pose
tree.
Metrics In order to be able to compare with published results we will use two widely accepted evaluation metrics.
Percentage of Correct Parts (PCP) measures detection rate
of limbs, where a limb is considered detected if the distance
between the two predicted joint locations and the true limb


Percentage of Correct Parts (PCP) measures detection rate
of limbs, where a limb is considered detected if the distance
between the two predicted joint locations and the true limb
joint locations is at most half of the limb length [5]. PCP
was the initially preferred metric for evaluation, however it
has the drawback of penalizing shorter limbs, such as lower
arms, which are usually harder to detect.
To address this drawback, recently detection rates of
joints are being reported using a different detection criterion – a joint is considered detected if the distance between
the predicted and the true joint is within a certain fraction of
the torso diameter. By varying this fraction, detection rates
are obtained for varying degrees of localization precision.
This metric alleviates the drawback of PCP since the detection criteria for all joints are based on the same distance
threshold. We refer to this metric as Percent of Detected
Joints (PDJ).
Experimental Details For all the experiments we use the
same network architecture. Inspired by [7], we use a body
detector on FLIC to obtain initially a rough estimate of the
human body bounding box. It is based on a face detector –
the detected face rectangle is enlarged by a ﬁxed scaler. This
scaler is determined on the training data such that it contains
all labeled joints. This face-based body detector results in


scaler is determined on the training data such that it contains
all labeled joints. This face-based body detector results in
a rough estimate, which however presents a good starting
point for our approach. For LSP we use the full image as
initial bounding box since the humans are relatively tightly
cropped by design.
Using a small held-out set of 50 images for both datasets
to determine the algorithm hyperparameters. To measure
optimality of the parameters we used average over PDJ at
0:2across all joints. The scaler , which deﬁnes the size of
the reﬁnement joint bounding box as a fraction of the pose
size, is determined as follows: for FLIC we chose = 1:0
after exploring values f0:8;1:0;1:2g, for LSP we use =
2:0after tryingf1:5;1:7;2:0;2:3g. The number of cascade
stagesSis determined by training stages until the algorithm
stopped improving on the held-out set. For both FLIC and
LSP we arrived at S= 3.
To improve generalization, for each cascade stage starting ats= 2 we augment the training data by sampling 40
randomly translated crop boxes for each joint as explained


To improve generalization, for each cascade stage starting ats= 2 we augment the training data by sampling 40
randomly translated crop boxes for each joint as explained
in Sec. 3.2. Thus, for LSP with 14joints and after mirroring the images and sampling the number training examples
is1100040214 = 12M, which is essential for
training a large network as ours.The presented algorithm allows for an efﬁcient implementation. The running time is approx. 0:1sper image,
as measured on a 12core CPU. This compares favorably
to other approaches, as some of the current state-of-art approaches have higher complexity: [20] runs in approx. 4s,
while [27] runs in 1:5s. The training complexity, however,
is higher. The initial stage was trained within 3days on
approx. 100 workers, most of the ﬁnal performance was
achieved after 12hours though. Each reﬁnement stage was
trained for 7days since the amount of data was 40larger
than the one for the initial stage due to the data augmentation in Sec. 3.2. Note that using more data led to increased
performance.
4.2. Results and Discussion


than the one for the initial stage due to the data augmentation in Sec. 3.2. Note that using more data led to increased
performance.
4.2. Results and Discussion
Comparisons We present comparative results to other approaches. We compare on LSP using PCP metric in Fig. 1.
We show results for the four most challenging limbs – lower
and upper arms and legs – as well as the average value
across these limbs for all compared algorithms. We clearly
outperform all other approaches, especially achieving better estimation for legs. For example, for upper legs we obtain0:78up from 0:74for the next best performing method.
It is worth noting that while the other approaches exhibit
strengths for particular limbs, none of the other dataset consistently dominates across all limbs. In contrary, DeepPose
shows strong results for all challenging limbs.
Using the PDJ metric allows us to vary the threshold for
the distance between prediction and ground truth, which deﬁnes a detection. This threshold can be thought of as a
localization precision at which detection rates are plotted.
Thus one could compare approaches across different desired precisions. We present results on FLIC in Fig. 3 comparing against additional four methods as well is on LSP in
Fig. 4. For each dataset we train and test according the protocol for each dataset. Similarly to previous experiment we


Fig. 4. For each dataset we train and test according the protocol for each dataset. Similarly to previous experiment we
outperform all ﬁve algorithms. Our gains are bigger in the
low precision domain, in the cases where we detect rough
pose without precisely localizing the joints. On FLIC, at
normalized distance 0:2we obtain a an increase of detection
rates by 0:15and0:2for elbow and wrists against the next
best performing method. On LSP, at normalized distance
0:5we get an absolute increase of 0:1. At low precision
regime of normalized distance of 0:2for LSP we show comparable performance for legs and slightly worse arms. This
can be attributed to the fact that the DNN-based approach
computes joint coordinates using 7 layers of transformation,
some of which contain max pooling.
Another observation is that our approach works well for
both appearance heavy movie data as well as string articulation such as the sports images in LSP.
Effects of cascade-based reﬁnement A single DNNbased joint regressor gives rough joint location. However,
0 0.05 0.1 0.15 0.20.10.30.50.70.9Elbows
Normalized distance to true jointDetection rate


0 0.05 0.1 0.15 0.20.10.30.50.70.9Elbows
Normalized distance to true jointDetection rate

DeepPoseMODECEichner et al.Yang et al.Sapp et al.
0 0.05 0.1 0.15 0.20.10.30.50.70.9Wrists
Normalized distance to true jointDetection rateFigure 3. Percentage of detected joints (PDJ) on FLIC for two joints: elbow and wrist. We compare DeepPose, after two cascade stages,
with four other approaches.
MethodArm LegAve.Upper Lower Upper Lower
DeepPose-st1 0.5 0.27 0.74 0.65 0.54
DeepPose-st2 0.56 0.36 0.78 0.70 0.60
DeepPose-st3 0.56 0.38 0.77 0.71 0.61
Dantone et al. [2] 0.45 0.25 0.65 0.61 0.49
Tian et al. [25]0.52 0.33 0.70 0.60 0.56
Johnson et al. [13] 0.54 0.38 0.75 0.66 0.58


Johnson et al. [13] 0.54 0.38 0.75 0.66 0.58
Wang et al. [26]0.565 0.37 0.76 0.68 0.59
Pishchulin [18]0.49 0.32 0.74 0.70 0.56
Table 1. Percentage of Correct Parts (PCP) at 0.5 on LSP for DeepPose as well as ﬁve state-of-art approaches.The authors use a
slightly looser version of PCP in which for each limb the average
distance from predicted limb joints belonging to the corresponding true joints is used to determine whether the limb is correctly
detected.The authors use person-centric joint annotations.
0 0.1 0.2 0.3 0.4 0.50.10.30.50.70.9Arms
Normalized distance to true jointDetection rate

DeepPose − wristsDeepPose − elbowsDantone et al. − wristsDantone et al. − elbows
0 0.1 0.2 0.3 0.4 0.50.10.30.50.70.9Legs
Normalized distance to true jointDetection rate

DeepPose − ankleDeepPose − kneeDantone et al. − ankleDantone et al. − knee


Normalized distance to true jointDetection rate

DeepPose − ankleDeepPose − kneeDantone et al. − ankleDantone et al. − knee
Figure 4. Percentage of detected joints (PDJ) on LSP for four
limbs for DeepPose and Dantone et al. [2] over an extended range
of distances to true joint: [0;0:5]of the torso diameter. Results of
DeepPose are plotted with solid lines while all the results by [2]
are plotted in dashed lines. Results for the same joint from both
algorithms are colored with same color.
to obtain higher precision the subsequent stages of the cascade, which serve as a reﬁnement of the initial prediction,
are of paramount importance. To see this, in Fig. 5 we
present the joint detections at different precisions for the initial prediction as well as two subsequent cascade stages. As
expected, we can see that the major gains of the reﬁnement
00.05 0.10.15 0.20.25 0.30.35 0.40.10.30.50.70.9Wrists
Normalized distance to true jointDetection rate

DeepPose − initial stage 1DeepPose − stage 2DeepPose − stage 3


Normalized distance to true jointDetection rate

DeepPose − initial stage 1DeepPose − stage 2DeepPose − stage 3
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.40.10.30.50.70.9Elbows
Normalized distance to true jointDetection rateFigure 5. Percent of detected joints (PDJ) on FLIC or the ﬁrst three
stages of the DNN cascade. We present results over larger spectrum of normalized distances between prediction and ground truth.
procedure are at high-precision regime of at normalized distances of [0:15;0:2]. Further, the major gains are achieved
after one stage of reﬁnement. The reason being that subsequent stages end up using smaller sub-images around each
joint. And although the subsequent stages look at higher
resolution inputs, they have more limited context.
Examples of cases, where reﬁnement helps, are visualized in Fig. 6. The initial stage is usually successful at estimating a roughly correct pose, however, this pose is not
”snapped” to the correct one. For example, in row three the
pose has the right shape but incorrect scale. In the second
row, the predicted pose is translated north from the ideal


pose has the right shape but incorrect scale. In the second
row, the predicted pose is translated north from the ideal
one. In most cases, the second stage of the cascade resolves
this snapping problem and better aligns the joints. In more
rare cases, such as in ﬁrst row, further facade stages improve
on individual joints.
Cross-dataset Generalization To evaluate the generalization properties of our algorithm, we used the trained
models on LSP and FLIC on two related datasets. The fullbody model trained on LSP is tested on the test portion of
the Image Parse dataset [19] with results presented in Table 2. The ImageParse dataset is similar to LSP as it contains people doing sports, however it contains a lot of people from personal photo collections involved in other activities. Further, the upper-body model trained on FLIC was
applied on the whole Buffy dataset [7]. We can see that our
Initial stage 1stage 2stage 3Figure 6. Predicted poses in red and ground truth poses in green
for the ﬁrst three stages of a cascade for three examples.
0 0.05 0.1 0.15 0.20.10.30.50.70.9Elbows
Normalized distance to true jointDetection rate

Eichner et al.Yang et al.Sapp et al.MODECDeepPose


Normalized distance to true jointDetection rate

Eichner et al.Yang et al.Sapp et al.MODECDeepPose
0 0.05 0.1 0.15 0.20.10.30.50.70.9Wrists
Normalized distance to true jointDetection rate
Figure 7. Percentage of detected joints (PDJ) on Buffy dataset
for two joints: elbow and wrist. The models have been trained on
FLIC. We compare DeepPose, after two cascade stages, with four
other approaches.
MethodArm LegAve.Upper Lower Upper Lower
DeepPose 0.8 0.75 0.71 0.5 0.69
Pishchulin [18] 0.80 0.70 0.59 037 0.62
Johnson et al. [13] 0.75 0.67 0.67 0.46 0.64
Yang et al. [27] 0.69 0.64 0.55 0.35 0.56
Table 2. Percentage of Correct Parts (PCP) at 0.5 on Image Parse
dataset for DeepPose as well as two state-of-art approaches on Image Parse dataset. Results obtained from [18].
approach can retain state-of-art performance compared to
other approaches. This shows good generalization abilities.


approach can retain state-of-art performance compared to
other approaches. This shows good generalization abilities.
Example poses To get a better idea of the performance of
our algorithm, we visualize a sample of estimated poses on
images from LSP in Fig. 8. We can see that our algorithm is
able to get correct pose for most of the joints under variety
of conditions: upside-down people (row 1, column 1), severe foreshortening (row1, column 3), unusual poses (row
3, column 5), occluded limbs as the occluded arms in row
3, columns 2 and 6, unusual illumination conditions (row 3,
column 3). In most of the cases, when the estimated pose isnot precise, it still has a correct shape. For example, in the
last row some of the predicted limbs are not aligned with
the true locations, however the overall shape of the pose is
correct. A common failure mode is confusing left with right
side when the person was photographed from the back (row
6, column 6). Results on FLIC (see Fig. 9) are usually better
with occasional visible mistakes on lower arms.
5. Conclusion
We present, to our knowledge, the ﬁrst application of
Deep Neural Networks (DNNs) to human pose estimation.
Our formulation of the problem as DNN-based regression to


Deep Neural Networks (DNNs) to human pose estimation.
Our formulation of the problem as DNN-based regression to
joint coordinates and the presented cascade of such regressors has the advantage of capturing context and reasoning
about pose in a holistic manner. As a result, we are able to
achieve state-of-art or better results on several challenging
academic datasets.
Further, we show that using a generic convolutional neural network, which was originally designed for classiﬁcation tasks, can be applied to the different task of localization. In future, we plan to investigate novel architectures
which could be potentially better tailored towards localization problems in general, and in pose estimation in particular.
Acknowledgements I would like to thank Luca Bertelli,
Ben Sapp and Tianli Yu for assistance with data and fruitful
discussions.
References
[1] M. Andriluka, S. Roth, and B. Schiele. Pictorial structures
revisited: People detection and articulated pose estimation.
InCVPR , 2009.
[2] M. Dantone, J. Gall, C. Leistner, and L. Van Gool. Human
pose estimation using body parts dependent joint regressors.
InCVPR , 2013.
[3] J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient


pose estimation using body parts dependent joint regressors.
InCVPR , 2013.
[3] J. Duchi, E. Hazan, and Y . Singer. Adaptive subgradient
methods for online learning and stochastic optimization. In
COLT . ACL, 2010.
[4] M. Eichner and V . Ferrari. Better appearance models for
pictorial structures. 2009.
[5] M. Eichner, M. Marin-Jimenez, A. Zisserman, and V . Ferrari.
Articulated human pose estimation and search in (almost)
unconstrained still images. ETH Zurich, D-ITET, BIWI,
Technical Report No , 272, 2010.
[6] P. F. Felzenszwalb and D. P. Huttenlocher. Pictorial structures for object recognition. International Journal of Computer Vision , 61(1):55–79, 2005.
[7] V . Ferrari, M. Marin-Jimenez, and A. Zisserman. Progressive
search space reduction for human pose estimation. In CVPR ,
2008.
[8] M. A. Fischler and R. A. Elschlager. The representation and
matching of pictorial structures. Computers, IEEE Transactions on , 100(1):67–92, 1973.


matching of pictorial structures. Computers, IEEE Transactions on , 100(1):67–92, 1973.
Figure 8. Visualization of pose results on images from LSP. Each pose is represented as a stick ﬁgure, inferred from predicted joints.
Different limbs in the same image are colored differently, same limb across different images has the same color.
Figure 9. Visualization of pose results on images from FLIC. Meaning of stick ﬁgures is the same as in Fig. 8 above.
[9] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014.
[10] G. Gkioxari, P. Arbel ´aez, L. Bourdev, and J. Malik. Articulated pose estimation using discriminative armlet classiﬁers.
InCVPR , 2013.
[11] C. Ionescu, F. Li, and C. Sminchisescu. Latent structured
models for human pose estimation. In ICCV , 2011.
[12] S. Johnson and M. Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC ,
2010.
[13] S. Johnson and M. Everingham. Learning effective human


2010.
[13] S. Johnson and M. Everingham. Learning effective human
pose estimation from inaccurate annotation. In CVPR , 2011.
[14] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classiﬁcation with deep convolutional neural networks. In NIPS ,
2012.
[15] G. Mori and J. Malik. Estimating human body conﬁgurations
using shape context matching. In ECCV , 2002.[16] R. Nevatia and T. O. Binford. Description and recognition of
curved objects. Artiﬁcial Intelligence , 8(1):77–98, 1977.
[17] M. Osadchy, Y . LeCun, and M. L. Miller. Synergistic face
detection and pose estimation with energy-based models.
The Journal of Machine Learning Research , 8:1197–1215,
2007.
[18] L. Pishchulin, M. Andriluka, P. Gehler, and B. Schiele. Poselet conditioned pictorial structures. In CVPR , 2013.
[19] D. Ramanan. Learning to parse images of articulated bodies.
InNIPS , 2006.


[19] D. Ramanan. Learning to parse images of articulated bodies.
InNIPS , 2006.
[20] B. Sapp and B. Taskar. Modec: Multimodal decomposable
models for human pose estimation. In CVPR , 2013.
[21] G. Shakhnarovich, P. Viola, and T. Darrell. Fast pose estimation with parameter-sensitive hashing. In CVPR , 2003.
[22] Y . Sun, X. Wang, and X. Tang. Deep convolutional network cascade for facial point detection. In Computer Vision
and Pattern Recognition (CVPR), 2013 IEEE Conference on ,
pages 3476–3483. IEEE, 2013.
[23] C. Szegedy, A. Toshev, and D. Erhan. Object detection via
deep neural networks. In NIPS 26 , 2013.
[24] G. W. Taylor, R. Fergus, G. Williams, I. Spiro, and C. Bregler. Pose-sensitive embedding by nonlinear nca regression.
InNIPS , 2010.
[25] Y . Tian, C. L. Zitnick, and S. G. Narasimhan. Exploring the
spatial hierarchy of mixture models for human pose estimation. In ECCV , 2012.


spatial hierarchy of mixture models for human pose estimation. In ECCV , 2012.
[26] F. Wang and Y . Li. Beyond physical connections: Tree models in human pose estimation. In CVPR , 2013.
[27] Y . Yang and D. Ramanan. Articulated pose estimation with
ﬂexible mixtures-of-parts. In CVPR , 2011.


1
Return of the Devil in the Details:
Delving Deep into Convolutional Nets
Ken Chatﬁeld, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman
Visual Geometry Group, Department of Engineering Science, University of Oxford
fken,karen,vedaldi,az g@robots.ox.ac.uk
Abstract —The latest generation of Convolutional Neural Networks (CNN) have achieved impressive results in challenging benchmarks on image recognition and object detection, signiﬁcantly raising the interest of the community in
these methods. Nevertheless, it is still unclear how different CNN methods compare with each other and with previous
state-of-the-art shallow representations such as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper
conducts a rigorous evaluation of these new techniques, exploring different deep architectures and comparing them on
a common ground, identifying and disclosing important implementation details. We identify several useful properties
of CNN-based representations, including the fact that the dimensionality of the CNN output layer can be reduced
signiﬁcantly without having an adverse effect on performance. We also identify aspects of deep and shallow methods
that can be successfully shared. In particular, we show that the data augmentation techniques commonly applied to
CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost. Source

Summary:
The latest generation of Convolutional Neural Networks (CNN) have achieved
impressive results in challenging benchmarks on image recognition and object
detection, significantly raising the interest of the community in these
methods. Nevertheless, it is still unclear how different CNN methods compare
with each other and with previous state-of-the-art shallow representations such
as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper conducts
a rigorous evaluation of these new techniques, exploring different deep
architectures and comparing them on a common ground, identifying and disclosing
important implementation details. We identify several useful properties of
CNN-based representations, including the fact that the dimensionality of the
CNN output layer can be reduced significantly without having an adverse effect
on performance. We also identify aspects of deep and shallow methods that can
be successfully shared. In particular, we show that the data augmentation
techniques commonly applied to CNN-based methods can also be applied to shallow
methods, and result in an analogous performance boost. Source code and models
to reproduce the experiments in the paper is made publicly available.

Ken Chatfield
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman

Published in proceedings of BMVC 2014


CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost. Source
code and models to reproduce the experiments in the paper is made publicly available.
F
1 I NTRODUCTION
PERHAPS the single most important design
choice in current state-of-the-art image classiﬁcation and object recognition systems is the choice
of visual features, or image representation. In fact,
most of the quantitative improvements to image
understanding obtained in the past dozen years can
be ascribed to the introduction of improved representations, from the Bag-of-Visual-Words (BoVW) [1],
[2] to the (Improved) Fisher Vector (IFV) [3]. A
common characteristic of these methods is that
they are largely handcrafted . They are also relatively
simple, comprising dense sampling of local image
patches, describing them by means of visual descriptors such as SIFT, encoding them into a highdimensional representation, and then pooling over
the image. Recently, these handcrafted approaches
have been substantially outperformed by the introduction of the latest generation of Convolutional
Neural Networks (CNNs) [4] to the computer vision
ﬁeld. These networks have a substantially more
sophisticated structure than standard representations, comprising several layers of non-linear feature extractors, and are therefore said to be deep

Published in proceedings of BMVC 2014


ﬁeld. These networks have a substantially more
sophisticated structure than standard representations, comprising several layers of non-linear feature extractors, and are therefore said to be deep
(in contrast, classical representation will be referred
to as shallow ). Furthermore, while their structureis handcrafted, they contain a very large number
of parameters learnt from data. When applied to
standard image classiﬁcation and object detection
benchmark datasets such as ImageNet ILSVRC [5]
and PASCAL VOC [6] such networks have demonstrated excellent performance [7], [8], [9], [10], [11],
signiﬁcantly better than standard image encodings [12].
Despite these impressive results, it remains unclear how different deep architectures compare to
each other and to shallow computer vision methods such as IFV . Most papers did not test these
representations extensively on a common ground,
so a systematic evaluation of the effect of different design and implementation choices remains
largely missing. As noted in our previous work [12],
which compared the performance of various shallow visual encodings, the performance of computer
vision systems depends signiﬁcantly on implementation
details . For example, state-of-the-art methods such
as [13] not only involve the use of a CNN, but
also include other improvements such as the use

Published in proceedings of BMVC 2014


details . For example, state-of-the-art methods such
as [13] not only involve the use of a CNN, but
also include other improvements such as the use
of very large scale datasets, GPU computation, and
data augmentation (also known as data jittering or
virtual sampling). These improvements could also
transfer to shallow representations such as the IFV ,
potentially explaining a part of the performancearXiv:1405.3531v4  [cs.CV]  5 Nov 2014
2
gap [14].
In this study we analyse and empirically clarify
these issues, conducting a large set of rigorous
experiments (Sect. 4), in many ways picking up
the story where it last ended in [12] with the
comparison of shallow encoders. We focus on methods to construct image representations ,i.e. encoding functions mapping an image Ito a vector
(I)2Rdsuitable for analysis with a linear classiﬁer, such as an SVM. We consider three scenarios
(Sect. 2, Sect. 3): shallow image representations,
deep representations pre-trained on outside data,
and deep representation pre-trained and then ﬁnetuned on the target dataset. As part of our tests,
we explore generally-applicable best practices that

Published in proceedings of BMVC 2014


and deep representation pre-trained and then ﬁnetuned on the target dataset. As part of our tests,
we explore generally-applicable best practices that
are nevertheless more often found in combination with CNNs [13] or, alternatively, with shallow
encoders [12], porting them with mutual beneﬁt.
These are (Sect. 2): the use of colour information ,
feature normalisation , and, most importantly, the
use of substantial data augmentation . We also determine scenario-speciﬁc best-practices , improving
the ones in [12], [15] and others, including dimensionality reduction for deep features. Finally, we
achieve performance competitive with the state
of the art [16], [17] on PASCAL VOC classiﬁcation
using less additional training data and signiﬁcantly
simpler techniques. As in [12], the source code and
models to reproduce all experiments in this paper
is available on the project website1.
2 S CENARIOS
This section introduces the three types of image
representation (I)considered in this paper, describing them within the context of three different
scenarios. Having outlined details speciﬁc to each,
general methodologies which apply to all three
scenarios are reviewed, such as data augmentation

Published in proceedings of BMVC 2014


scenarios. Having outlined details speciﬁc to each,
general methodologies which apply to all three
scenarios are reviewed, such as data augmentation
and feature normalisation, together with the linear
classiﬁer (trained with a standard hinge loss). We
also specify here the benchmark datasets used in
the evaluation.
2.1 Scenario 1: Shallow representation (IFV)
Our reference shallow image representation is the
IFV [3]. Our choice is motivated by the fact that
IFV usually outperforms related encoding methods
such as BoVW, LLC [12], and VLAD [18]. Given an
imageI, the IFVFV(I)is obtained by extracting
1. http://www.robots.ox.ac.uk/vgg/research/deep eval/a dense collection of patches and corresponding
local descriptors xi2RD(e.g. SIFT) from the image
at multiple scales. Each descriptor xiis then softquantized using a Gaussian Mixture Model with
Kcomponents. First and second order differences
between each descriptor xiand its Gaussian cluster
meankare accumulated in corresponding blocks
uk,vkin the vector FV(I)2R2KD, appropriately
weighed by the Gaussian soft-assignments and

Published in proceedings of BMVC 2014


uk,vkin the vector FV(I)2R2KD, appropriately
weighed by the Gaussian soft-assignments and
covariance, leading to a 2KD-dimensional image
representation FV(I) = [u>
1;v>
1;:::u>
K;v>
K]>:The
improved version of the Fisher vector involves postprocessing FVby computing the signed squareroot of its scalar components and normalising the
result to a unit `2norm. The details of this construction can be found in [3]; here we follow the
notation of [12].
2.2 Scenario 2: Deep representation (CNN) with
pre-training
Our deep representations are inspired by the success of the CNN of Krizhevsky et al. [13]. As shown
in [7], [19], the vector of activities CNN(I)of the
penultimate layer of a deep CNN, learnt on a large
dataset such as ImageNet [5], can be used as a powerful image descriptor applicable to other datasets.
Numerous CNN architectures that improve the
previous state of the art obtained using shallow
representations have been proposed, but choosing

Published in proceedings of BMVC 2014


Numerous CNN architectures that improve the
previous state of the art obtained using shallow
representations have been proposed, but choosing
the best one remains an open question. Many are inspired by [13]: DeCAF [7], [11], Caffe [20], Oquab et
al. [8]. Others use larger networks with a smaller
stride of the ﬁrst convolutional layer: Zeiler and
Fergus [19] and OverFeat [10], [9]. Other differences
include the CNN pre-training protocols. Here we
adopt a single learning framework and experiment
with architectures of different complexity exploring
their performance-speed trade-off.
2.3 Scenario 3: Deep representation (CNN) with
pre-training and ﬁne-tuning
In Scenario 2 features are trained on one (large)
dataset and applied to another (usually smaller).
However, it was demonstrated [11] that ﬁne-tuning
a pre-trained CNN on the target data can signiﬁcantly improve the performance. We consider
this scenario separately from that of Scenario 2, as
the image features become dataset-speciﬁc after the
ﬁne-tuning.
3
2.4 Commonalities
We now turn to what is in common across the
scenarios.

Published in proceedings of BMVC 2014


ﬁne-tuning.
3
2.4 Commonalities
We now turn to what is in common across the
scenarios.
2.4.1 Data augmentation
Data augmentation is a method applicable to shallow and deep representations, but that has been
so far mostly applied to the latter [13], [19]. By
augmentation we mean perturbing an image Iby
transformations that leave the underlying class unchanged ( e.g. cropping and ﬂipping) in order to
generate additional examples of the class. Augmentation can be applied at training time, at test time,
or both. The augmented samples can either be taken
as-is or combined to form a single feature, e.g. using
sum/max-pooling or stacking.
2.4.2 Linear predictors
All the representations (I)in the three scenarios
are used to construct linear predictorshw;(I)ifor
each class to be recognized. These predictors are
learnt using Support Vector Machines (SVM) by
ﬁtting wto the available training data by minimizing an objective function balancing a quadratic
regularizer and the hinge-loss. The parameter C
in the SVM, trading-off regularizer and loss, is
determined using an held-off validation subset of

Published in proceedings of BMVC 2014


regularizer and the hinge-loss. The parameter C
in the SVM, trading-off regularizer and loss, is
determined using an held-off validation subset of
the data. Here we use the same learning framework
with all representations. It is common experience
that linear classiﬁers are particularly sensitive to
thenormalisation of the data and that, in particular,
SVMs tend to beneﬁt from `2normalisation [3] (an
interpretation is that after normalisation the inner
product corresponds to the cosine similarly).
2.5 Benchmark data
As reference benchmark we use the PASCAL
VOC [6] data as already done in [12]. The VOC2007 edition contains about 10,000 images split into
train, validation, and test sets, and labelled with
twenty object classes. A one-vs-rest SVM classiﬁer
for each class is learnt and evaluated independently
and the performance is measured as mean Average
Precision (mAP) across all classes. The VOC-2012
edition contains roughly twice as many images
and does not include test labels; instead, evaluation
uses the ofﬁcial PASCAL Evaluation Server. To train
deep representations we use the ILSVRC-2012 challenge dataset. This contains 1,000 object categories

Published in proceedings of BMVC 2014


uses the ofﬁcial PASCAL Evaluation Server. To train
deep representations we use the ILSVRC-2012 challenge dataset. This contains 1,000 object categories
from ImageNet [5] with roughly 1.2M training images, 50,000 validation images, and 100,000 testimages. Performance is evaluated using the top-5
classiﬁcation error. Finally, we also evaluate over
theCaltech-101 and Caltech-256 image classiﬁcation benchmarks [21], [22]. For Caltech-101, we
followed the protocol of [12], and considered three
random splits into training and testing data, each
of which comprises 30 training and up to 30 testing
images per class. For Caltech-256, two random
splits were generated, each of which contains 60
training images per class, and the rest are used for
testing. On both Caltech datasets, performance is
measured using mean class accuracy.
3 D ETAILS
This section gives the implementation details of the
methods introduced in Sect. 2.
3.1 Improved Fisher Vector details
Our IFV representation uses a slightly improved
setting compared to the best result of [12].
Computation starts by upscaling the image Iby a
factor of 2 [23], followed by SIFT features extraction

Published in proceedings of BMVC 2014


setting compared to the best result of [12].
Computation starts by upscaling the image Iby a
factor of 2 [23], followed by SIFT features extraction
with a stride of 3 pixels at 7 different scales withp
2
scale increments. These features are square-rooted
as suggested by [24], and decorrelated and reduced
in dimension from 128Dto80Dusing PCA. A
GMM with K= 256 components is learnt from features sampled from the training images. Hence the
Fisher Vector FV(I)has dimension 2KD= 40;960.
Before use in classiﬁcation, the vector is signedsquare-rooted and l2-normalised (square rooting
correspond to the Hellinger’s kernel map [25]). As
in [12], square-rooting is applied twice, once to the
raw encodings, and once again after sum pooling
and normalisation. In order to capture weak geometrical information, the IFV representation is used
in a spatial pyramid [26]. As in [12], the image is
divided into 11,31, and 22spatial subdivisions
and corresponding IFVs are computed and stacked
with an overall dimension of 82KD= 327;680
elements.

Published in proceedings of BMVC 2014


and corresponding IFVs are computed and stacked
with an overall dimension of 82KD= 327;680
elements.
In addition to this standard formulation, we experiment with a few modiﬁcations. The ﬁrst one
is the use of intra-normalisation of the descriptor
blocks, an idea recently proposed for the VLAD
descriptor [27]. In this case, the `2normalisation
is applied to the individual sub-blocks (uk;vk)of
the vector FV(I), which helps to alleviate the
local feature burstiness [28]. In the case of the
improved intra-normalised features, it was found
4
that applying the square-rooting only once to the
ﬁnal encoding produced the best results.
The second modiﬁcation is the use of spatiallyextended local descriptors [23] instead of a spatial
pyramid. Here descriptors xiare appended with
their image location (xi;yi)before quantization
with the GMM. Formally, xiis extended, after PCA
projection, with its normalised spatial coordinates:
[x>
i;xi=W0:5;yi=H0:5]>, whereWHare the
dimensions of the image. Since the GMM quantizes

Published in proceedings of BMVC 2014


i;xi=W0:5;yi=H0:5]>, whereWHare the
dimensions of the image. Since the GMM quantizes
both appearance and location, this allows for spatial
information to be captured directly by the softquantization process. This method is signiﬁcantly
more memory-efﬁcient than using a spatial pyramid. Speciﬁcally, the PCA-reduced SIFT features
are spatially augmented by appending (x;y)yieldingD= 82 dimensional descriptors pooled in a
2KD= 41;984dimensional IFV .
The third modiﬁcation is the use of colour features in addition to SIFT descriptors. While colour
information is used in CNNs [13] and by the original FV paper [3], it was not explored in our previous comparison [12]. We do so here by adopting
the same Local Colour Statistics (LCS) features as
used by [3]. LCS is computed by dividing an input
patch into a 44spatial grid (akin to SIFT), and
computing the mean and variance of each of the
Lab colour channels for each cell of the grid. The
LCS dimensionality is thus 4423 = 96 . This
is then encoded in a similar manner to SIFT.

Published in proceedings of BMVC 2014


Lab colour channels for each cell of the grid. The
LCS dimensionality is thus 4423 = 96 . This
is then encoded in a similar manner to SIFT.
3.2 Convolutional neural networks details
The CNN-based features are based on three CNN
architectures representative of the state of the art
(shown in Table 1) each exploring a different accuracy/speed trade-off. To ensure a fair comparison
between them, these networks are trained using the
same training protocol and the same implementation, which we developed based on the opensource Caffe framework [20]. `2-normalising the
CNN features CNN(I)before use in the SVM was
found to be important for performance.
Our Fast (CNN-F) architecture is similar to the
one used by Krizhevsky et al . [13]. It comprises
8 learnable layers, 5 of which are convolutional,
and the last 3 are fully-connected. The input image
size is 224224. Fast processing is ensured by the
4 pixel stride in the ﬁrst convolutional layer. The
main differences between our architecture and that
of [13] are the reduced number of convolutional
layers and the dense connectivity between convolu-tional layers ([13] used sparse connections to enable
training on two GPUs).

Published in proceedings of BMVC 2014


of [13] are the reduced number of convolutional
layers and the dense connectivity between convolu-tional layers ([13] used sparse connections to enable
training on two GPUs).
Our Medium (CNN-M) architecture is similar
to the one used by Zeiler and Fergus [19]. It is
characterised by the decreased stride and smaller
receptive ﬁeld of the ﬁrst convolutional layer, which
was shown to be beneﬁcial on the ILSVRC dataset.
At the same time, conv2 uses larger stride (2 instead
of 1) to keep the computation time reasonable. The
main difference between our net and that of [19] is
we use less ﬁlters in the conv4 layer (512 vs. 1024).
Our Slow (CNN-S) architecture is related to the
‘accurate’ network from the OverFeat package [10].
It also uses 77ﬁlters with stride 2in conv1.
Unlike CNN-M and [19], the stride in conv2 is
smaller (1 pixel), but the max-pooling window in
conv1 and conv5 is larger ( 33) to compensate for
the increased spatial resolution. Compared to [10],
we use 5 convolutional layers as in the previous

Published in proceedings of BMVC 2014