High-Performance Neural Networks
for Visual Object Classication
Dan C. Cire san, Ueli Meier, Jonathan Masci,
Luca M. Gambardella and J urgen Schmidhuber
Technical Report No. IDSIA-01-11
January 2011
IDSIA / USI-SUPSI
Dalle Molle Institute for Articial Intelligence
Galleria 2, 6928 Manno, Switzerland
IDSIA is a joint institute of both University of Lugano (USI) and University of Applied Sciences of Southern Switzerland (SUPSI),
and was founded in 1988 by the Dalle Molle Foundation which promoted quality of life.
This work was partially supported by the Swiss Commission for Technology and Innovation (CTI), Project n. 9688.1 IFF:
Intelligent Fill in Form.arXiv:1102.0183v1  [cs.AI]  1 Feb 2011
Technical Report No. IDSIA-01-11 1
High-Performance Neural Networks
for Visual Object Classication
Dan C. Cire san, Ueli Meier, Jonathan Masci,
Luca M. Gambardella and J urgen Schmidhuber
January 2011
Abstract
We present a fast, fully parameterizable GPU implementation of Convolutional Neural
Network variants. Our feature extractors are neither carefully designed nor pre-wired, but

High-Performance Neural Networks for Visual Object Classification

Summary:
We present a fast, fully parameterizable GPU implementation of Convolutional
Neural Network variants. Our feature extractors are neither carefully designed
nor pre-wired, but rather learned in a supervised way. Our deep hierarchical
architectures achieve the best published results on benchmarks for object
classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with
error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple
back-propagation perform better than more shallow ones. Learning is
surprisingly rapid. NORB is completely trained within five epochs. Test error
rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs,
respectively.

Dan C. Cireşan
Ueli Meier
Jonathan Masci
Luca M. Gambardella
Jürgen Schmidhuber

12 pages, 2 figures, 5 tables


January 2011
Abstract
We present a fast, fully parameterizable GPU implementation of Convolutional Neural
Network variants. Our feature extractors are neither carefully designed nor pre-wired, but
rather learned in a supervised way. Our deep hierarchical architectures achieve the best
published results on benchmarks for object classication (NORB, CIFAR10) and handwritten
digit recognition (MNIST), with error rates of 2.53%, 19.51%, 0.35%, respectively. Deep
nets trained by simple back-propagation perform better than more shallow ones. Learning
is surprisingly rapid. NORB is completely trained within ve epochs. Test error rates on
MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs, respectively.
1 Introduction
The human visual system eciently recognizes and localizes objects within cluttered scenes. For
articial systems, however, this is still dicult, due to viewpoint-dependent object variability,
and the high in-class variability of many object types. Deep hierarchical neural models roughly
mimick the nature of mammalian visual cortex, and by community consensus are among the most
promising architectures for such tasks. The most successful hierarchical object recognition systems
all extract localized features from input images, convolving image patches with lters. Filter

12 pages, 2 figures, 5 tables


promising architectures for such tasks. The most successful hierarchical object recognition systems
all extract localized features from input images, convolving image patches with lters. Filter
responses are then repeatedly sub-sampled and re-ltered, resulting in a deep feed-forward network
architecture whose output feature vectors are eventually classied. One of the rst hierarchical
neural systems was the Neocognitron (Fukushima, 1980) which inspired many of the more recent
variants.
Unsupervised learning methods applied to patches of natural images tend to produce localized
lters that resemble o-center-on-surround lters, orientation-sensitive bar detectors, Gabor lters
(Schmidhuber et al. , 1996; Olshausen and Field, 1997; Hoyer and Hyv arinen, 2000). These ndings
in conjunction with experimental studies of the visual cortex justify the use of such lters in the
so-called standard model for object recognition (Riesenhuber and Poggio, 1999; Serre et al. , 2007;
Mutch and Lowe, 2008), whose lters are xed, in contrast to those of Convolutional Neural

12 pages, 2 figures, 5 tables


Mutch and Lowe, 2008), whose lters are xed, in contrast to those of Convolutional Neural
Networks (CNNs) (LeCun et al. , 1998; Behnke, 2003; Simard et al. , 2003), whose weights (lters)
are randomly initialized and changed in a supervised way using back-propagation (BP).
Despite the hardware progress of the past decades, computational speed is still a limiting
factor for CNN architectures characterized by many building blocks typically set by trial and
error. To systematically test the impact of various architectures on classication performance,
we present a fast CNN implementation on Graphics Processing Units (GPUs). Previous GPU
implementations of CNNs (Chellapilla et al. , 2006; Uetz and Behnke, 2009) were hard-coded to
satisfy GPU hardware constraints, whereas our implementation is
exible and fully online (i.e.,
Technical Report No. IDSIA-01-11 2
weight updates after each image). It allows for training large CNNs within days instead of months,
such that we can investigate the in
uence of various structural parameters by exploring large
parameter spaces (Pinto et al. , 2009) and performing error analysis on repeated experiments.
We evaluate various networks on the handwritten digit benchmark MNIST (LeCun et al. , 1998)

12 pages, 2 figures, 5 tables


We evaluate various networks on the handwritten digit benchmark MNIST (LeCun et al. , 1998)
and two image classication benchmarks: NORB (LeCun et al. , 2004) and CIFAR10 (Krizhevsky,
2009).
2 Convolutional neural networks
CNNs are hierarchical neural networks whose convolutional layers alternate with subsampling
layers, reminiscent of simple and complex cells in the primary visual cortex (Wiesel and Hubel,
1959). CNNs vary in how convolutional and subsampling layers are realized and how the nets
are trained. The CNN architecture considered in this study diers from the one of Simard et al.
(2003) in the sense that after each CNN-layer an optional max-pooling layer (Scherer et al. , 2010)
can be used. Here we give a complete description of this independent implementation (Fig. 1).
2.1 Image processing layer
The image processing layer is an optional pre-processing layer of predened lters that are kept
xed during training. Thus additional information besides the raw input image can be provided
to the network, such as edges and gradients. In particular, we nd that a contrast-extracting layer
(Fukushima, 2003) helps to improve the recognition rate for NORB.
2.2 Convolutional layer

12 pages, 2 figures, 5 tables


(Fukushima, 2003) helps to improve the recognition rate for NORB.
2.2 Convolutional layer
A convolutional layer is parametrized by the size and the number of the maps, kernel sizes, skipping
factors, and the connection table. Each layer has Mmaps of equal size ( Mx,My). A kernel (blue
rectangle in Fig 1) of size ( Kx,Ky) is shifted over the valid region of the input image (i.e. the
kernel has to be completely inside the image). The skipping factors SxandSydene how many
pixels the lter/kernel skips in x- and y-direction between subsequent convolutions. The size of
the output map is then dened as:
Mn
x=Mn1
xKn
x
Snx+ 1+ 1;Mn
y=Mn1
yKn
y
Sny+ 1+ 1 (1)
where index nindicates the layer. Each map in layer Lnis connected to at most Mn1maps in
layerLn1. Neurons of a given map share their weights but have dierent receptive elds.
2.3 Max-pooling layer
The biggest architectural dierence between our implementation and the CNN of LeCun et al.

12 pages, 2 figures, 5 tables


2.3 Max-pooling layer
The biggest architectural dierence between our implementation and the CNN of LeCun et al.
(1998) is the use of a max-pooling layer instead of a sub-sampling layer. No such layer is used
by Simard et al. (2003) who simply skips nearby pixels prior to convolution, instead of pooling
or averaging. Scherer et al. (2010) found that max-pooling can lead to faster convergence, select
superior invariant features, and improve generalization. The output of the max-pooling layer is
given by the maximum activation over non-overlapping rectangular regions of size ( Kx,Ky). Maxpooling enables position invariance over larger local regions and downsamples the input image by
a factor ofKxandKyalong each direction.
Technical Report No. IDSIA-01-11 3
2.4 Classication layer
Kernel sizes of convolutional lters and max-pooling rectangles as well as skipping factors are
chosen such that either the output maps of the last convolutional layer are downsampled to 1
pixel per map, or a fully connected layer combines the outputs of the topmost convolutional layer
into a 1D feature vector. The top layer is always fully connected, with one output unit per class
label.

12 pages, 2 figures, 5 tables


into a 1D feature vector. The top layer is always fully connected, with one output unit per class
label.
Figure 1: Architecture of a convolutional neural network. In this case, the convolutional layers
are fully connected. Both convolutional layers use a kernel of 5 x 5 and skipping factors of 1.
3 GPU implementation
The latest generation of NVIDIA GPUs, the 400 and 500 series (we use GTX 480 & GTX 580), has
many advantages over older GPUs, most notably the presence of a R/W L2 global cache for device
memory. This permits faster programs and simplies writing the code. In fact, the corresponding
transfer of complexity into hardware alleviates many software and optimization problems. Our
experiments show that the CNN program becomes 2-3 times faster just by switching from GTX
285 to GTX 480.
Technical Report No. IDSIA-01-11 4
Manual optimization of CUDA code is very time-consuming and error prone. We optimize for
the new architecture, relying on the L2 cache for many of the device memory accesses, instead of
manually writing code that uses textures and shared memory. Code obtained by this pragmatic
strategy is fast enough. We use the following types of optimization: pre-computed expressions,
unrolled loops within template kernels, strided matrices to obtain coalesced memory accesses and

12 pages, 2 figures, 5 tables


strategy is fast enough. We use the following types of optimization: pre-computed expressions,
unrolled loops within template kernels, strided matrices to obtain coalesced memory accesses and
registers wherever possible. Additional manual optimizations are possible in case future image
classication problems will require even more computing power.
3.1 Data structures
Both outputs yand deltasof layerLnare 2D strided. Their original size is MxMM y, but they
are horizontally strided with a pitch of 32
oats (we use this stride for all 2D data), resulting in
coalesced memory accesses. The vertical stride avoids additional bounding tests in CUDA kernels.
All connections between maps of consecutive layers Ln1andLnare stored in matrix Cn.
Each row of Cncontains all connections that feed into a particular map in layer Ln. Because
we aim for a
exible architecture with partially connected layers, in the rst column we store the
number of previous connections. This index is useful for Forward Propagation (FP) and Adjusting
Weights (AW) CUDA kernels. The second column stores the number of connections, followed by
corresponding indices of maps in Ln1connected to the current map.

12 pages, 2 figures, 5 tables


Weights (AW) CUDA kernels. The second column stores the number of connections, followed by
corresponding indices of maps in Ln1connected to the current map.
For BP and FP, analogous information about connections is needed. We therefore store backward connections in CBP. AW requires a list of all map connections (see Subsection 3.4), stored
as an array of map index pairs. Dealing with biases in BP kernel requires to know where the
weights of particular connections start; this information is stored in a 2D array WIDX BPof size
MnMn1.
3.2 Forward propagation
A straightforward way of parallelizing FP is to assign a thread block to each map that has to
be computed. For maps bigger than 1024 neurons, the job is further split into smaller blocks by
assigning a block to each line of the map, because the number of threads per block is limited (1024
for GTX 480). A one to one correspondence between threads and the map's neurons is assumed.
Because of weight sharing, threads inside a block can access data in parallel, in particular the
same weights and inputs from the previous layer. Each thread starts by initializing its sum with
the bias, then loops over all map connections, convolving the appropriate patch of the input map
with the corresponding kernel. The output is obtained by passing the sum through a scaled tanh

12 pages, 2 figures, 5 tables


the bias, then loops over all map connections, convolving the appropriate patch of the input map
with the corresponding kernel. The output is obtained by passing the sum through a scaled tanh
activation function, and then written to device memory.
3.3 Backward propagation
BP of deltas can be done in two ways: by pushing or by pulling. Pushing deltas means taking each
delta from the current layer and computing the corresponding deltas for the previous layer. For
an architecture with shared weights this has the disadvantage of being hard to code. Each delta
from the current layer contributes to many deltas in the previous layer, which translates into a lot
of programming. There are two ways of avoiding this: either writing partial deltas to a separated
block of memory and then putting everything together by calling another kernel (slow because
of a tremendous increase in the number of memory accesses, and the need of another kernel), or
using atomic writes (to avoid data hazards) to update deltas (very slow because many writings
are serialized). We implement pulling deltas, which has almost none of the above speed-limiting
drawbacks, but is a bit more complicated.
Technical Report No. IDSIA-01-11 5
The (uni- or bi-dimensional) thread grid assigns a (bi- or uni-dimensional) thread block to each

12 pages, 2 figures, 5 tables


Technical Report No. IDSIA-01-11 5
The (uni- or bi-dimensional) thread grid assigns a (bi- or uni-dimensional) thread block to each
map in the previous layer and a thread to each neuron in every map. Similar to FP, for maps
with more than 1024 neurons, the 2D grid is further split into smaller 1D blocks by assigning a
2D block to each row of the map. Each thread computes the delta of its corresponding neuron by
pulling deltas from the current layer. For every neuron in the previous layer we have to determine
the list of neurons in the current layer which are connected to it. Let us consider neuron ( i;j)
from a map in layer Ln1, and then assume that ( x;y) are the coordinates of neurons in maps
ofLnthat contribute to the delta of neuron ( i;j). The (x;y) neuron is connected to kernel size
number neurons ( KxKy) from each connected map in the previous layer. The indices in Ln1
of the neurons connected through a kernel to the ( x;y) neuron are:
x(Sx+ 1) ix(Sx+ 1) +Kx1;
y(Sy+ 1) jy(Sy+ 1) +Ky1:

12 pages, 2 figures, 5 tables


y(Sy+ 1) jy(Sy+ 1) +Ky1:
We can now compute the inequalities for ( x;y):
iKx+ 1
Sx+ 1xi
Sx+ 1;
jKy+ 1
Sy+ 1yj
Sy+ 1:
Because (x;y) has to be inside the map, the nal inequalities are:
maxiKx+ 1
Sx+ 1
;0
xmini
Sx+ 1
;Mx1
;
maxjKy+ 1
Sy+ 1
;0
yminj
Sy+ 1
;My1
:
The above inequalities state that the delta of neuron ( i;j) fromLn1is computed from deltas of
neurons in a rectangular area in maps of Ln(Fig. 2). After summing up the deltas, each thread
multiplies the result by the derivative of the activation function.
3.4 Adjusting weights
FP and BP have a grid on the list of maps, but the AW thread grid is on the list of kernels (lters)

12 pages, 2 figures, 5 tables


3.4 Adjusting weights
FP and BP have a grid on the list of maps, but the AW thread grid is on the list of kernels (lters)
between maps of two consecutive layers. The 1D grid has a block for each connection between two
maps. Thread blocks are 2D, with a corresponding thread for every kernel weight. The bias weight
is included as an entire row of threads, thus requiring thread blocks to have ( Kx+1)Kythreads.
Most of the time these additional Kythreads will do nothing, thread (0,0) being activated only
for blocks that have to process the bias.
4 Experiments
We use a system with a Core i7-920 (2.66GHz), 12 GB DDR3 and four graphics cards: 2 x
GTX 480 and 2 x GTX 580. The correctness of the CPU version is checked by comparing the
analytical gradient with its nite dierence approximation. On GPU this is not possible because
all computations are performed with single precision
oating point numbers. Hence the GPU
implementation's correctness is checked by comparing its results to those of a randomly initialized
net after training it for several epochs on the more accurate CPU version. Obtaining identical
results after trillions of operations is a strong indication of correctness.
Technical Report No. IDSIA-01-11 6

12 pages, 2 figures, 5 tables


results after trillions of operations is a strong indication of correctness.
Technical Report No. IDSIA-01-11 6
Figure 2: Back propagating deltas. A connection between two maps from two consecutive layers
is displayed. The map in Ln1has 29 x 29 neurons; the map in Lnhas 13 x 13 neurons. They
are linked through a 5 x 5 kernel K. Skipping factors of Sx= 1 andSy= 1 are assumed. Arrows
and colors depict the correspondence between neurons in Ln1and their sources in Ln.
The implemented CNN's plain feed-forward architecture is trained using on-line gradient descent. All images from the training set are used for training and also for validation. If deformations
are enabled, only the images from the training set will be deformed. Weights are initialized according to a uniform random distribution in the range [ 0:05;0:05]. Each neuron's activation
function is a scaled hyperbolic tangent: y(a) = 1:7159 tanh(0 :6666a) (LeCun et al. , 1998).
We pick the trained CNN with the lowest validation error, and evaluate it on the test set
(Test for best Validation - TfbV). The best test error (bT) is also listed for all experiments. The

12 pages, 2 figures, 5 tables


(Test for best Validation - TfbV). The best test error (bT) is also listed for all experiments. The
reported computation times per epoch include training, validation and testing as well as all data
transfers.
4.1 Experiments on MNIST
For the MNIST dataset the networks are trained on deformed images, continually generated in
on-line fashion. Ane (translation, rotation, scaling, horizontal shearing) and elastic deformations
(Simard et al. , 2003) are combined. We use a variable learning rate that shrinks by a multiplicative
constant after each epoch, from 103down to 3 105after 500 epochs.
Fully connected convolutional layers lead to an exploding number of network connections and
weights, making training of big and deep CNNs for hundreds of epochs impractical even on GPUs.
Partial connectivity alleviates this problem and is also biologically more plausible. We reduce
the number of connections between convolutional layers in a random way. Table 1 lists results
of various networks with 2 to 7 hidden layers with random connections. Additional layers result
in better networks, the best one achieving a test error of 0.35% for best validation and a best
test error of 0.27%. The best previous CNN result on MNIST is 0.40% (Simard et al. , 2003).

12 pages, 2 figures, 5 tables


test error of 0.27%. The best previous CNN result on MNIST is 0.40% (Simard et al. , 2003).
A 0.35% error rate was recently also obtained by a big, deep MLP (Cire san et al. , 2010) with
many more free parameters. Deeper nets require more computation time to complete an epoch,
but we observe that they also need fewer epochs to achieve good test errors. The deepest CNN
Technical Report No. IDSIA-01-11 7
Table 1: Error rates on MNIST test set for randomly connected CNNs with 2 to 6 convolutional
layers with M Maps and an optional fully connected layer with N neurons. Various kernel sizes
and skipping factors were used.
#M, #N bT TfbV
in Hidden Layers [%] [%]
20M-60M 0.95 1.02
20M-60M-150N 0.50 0.55
20M-60M-100M-150N 0.33 0.38
20M-40M-60M-80M-100M-120M-150N 0.27 0.35
from Table 1 reaches 2.42%, 0.97% and 0.48% after one, three and seventeen epochs, respectively.

12 pages, 2 figures, 5 tables


from Table 1 reaches 2.42%, 0.97% and 0.48% after one, three and seventeen epochs, respectively.
On the other hand, the network with 4 instead of 7 hidden layers reaches 4.71%, 1.58%, 0.68%
after one, three and seventeen epochs, achieving a test error below 0.50% after only 34 epochs.
This shows once more that deep networks, contrary to common belief, can be trained successfully
by back-propagation. Despite the numerous free parameters, deep networks seem to learn faster
(better recognition rates after fewer epochs) than shallow ones.
We consider MNIST an almost solved problem. The remaining errors stem from digits that
are ambiguous or miss parts.
4.2 Experiments on NORB
NORB contains stereo images of 3D objects. Hence there are two maps on the input layer.
Rotation, scaling, shearing and elastic distortions seem to have a negative impact on generalization.
These deformations improve recognition rates for digits that are intrinsically 2D (Cire san et al. ,
2010), but seem inadequate for 3D objects.
Initial experiments on NORB show that unlike with MNIST where we use deformations, the
CNN needs only 3 to 6 epochs to reach zero validation error. This allows us to quickly run numerous

12 pages, 2 figures, 5 tables


Initial experiments on NORB show that unlike with MNIST where we use deformations, the
CNN needs only 3 to 6 epochs to reach zero validation error. This allows us to quickly run numerous
repetitive experiments with huge networks with hundreds of maps per layer. We decided to use a
CNN with ve hidden layers: layer1, a convolutional layer with 300 maps, kernel size 6 6 and
skipping factors 1 1; layer2, a max-pooling layer over a 2 2 region; layer3, a convolutional layer
with 500 maps, kernel size 4 4, skipping factors 0 0; layer4, a max-pooling layer over a 4 4
region; layer5, a fully connected layer with 500 neurons. The learning rate is initialized by 0.001
and multiplied by 0.95 after every epoch.
Table 2 summarizes the results of four dierent experiments by switching on/o translation as
well as the xed image processing layer. We report the average error rate as well as the standard
deviation of N independent runs with identical architectures but dierent weight initializations.
For the rst experiment without translation and no image processing (IP), an average test error
rate of 7.86% is obtained. With additional translations of at most 5%, the average error rate drops

12 pages, 2 figures, 5 tables


rate of 7.86% is obtained. With additional translations of at most 5%, the average error rate drops
to 4.71%, contradicting the common belief that CNNs are translation invariant. These results are
on par or better than others in the literature: 5.90% error rate for a combination of CNNs and
SVMs (LeCun et al. , 2004) and 5.20% error rate for restricted Boltzman machines (Nair and
Hinton, 2009).
The best previously published result on NORB (2.87%) was obtained by a hierarchical neural
network which to every convolutional layer provides a subsampled version plus edge information
of the original image (Uetz and Behnke, 2009). This motivated us to implement a pre-processing
layer with xed lters. We tried simple edge masks (Sobel, Scharr) but obtained best results with a
contrast-extraction layer (Fukushima, 2003) realized by Mexican hat-shaped lters of size 21 21,
one with a concentric on-center receptive eld and one with a concentric o-center receptive eld,
Technical Report No. IDSIA-01-11 8
Table 2: Average error rates and standard deviations of N runs for a ve hidden layer CNN on

12 pages, 2 figures, 5 tables


Technical Report No. IDSIA-01-11 8
Table 2: Average error rates and standard deviations of N runs for a ve hidden layer CNN on
the NORB test set (see text for details).
trans. [%] IP TfbV [%] runs time/epoch [s]
0 no 7.86 0.55 50 1141
5 no 4.71 0.57 50 1563
0 yes 3.94 0.48 50 1658
5 yes 2.530.40 100 2080
similar to the lters automatically created by unsupervised Predictability Minimization (Schmidhuber, 1992) applied to natural images (Schmidhuber et al. , 1996). The rst lter extracts positive
contrast in brightness, whereas the latter extracts negative contrast. Each image from the original
NORB is ltered, consequently the input of the CNN has six maps: the original image plus the
positive and negative contrast for each of the two stereo channels. Using such a pre-processing
layer results in lower average error rates, 3.94% without translation and 2.53% with translation.
This result improves the previous state of the art on NORB (Uetz and Behnke, 2009).
Experience with other image datasets tells us that NORB is unusual. The training set has only

12 pages, 2 figures, 5 tables


Experience with other image datasets tells us that NORB is unusual. The training set has only
ve instances per class. The resulting poor training set variability makes the nets learn quickly
but generalize badly. NORB is the only dataset that prots from a xed pre-processing layer in
a substantial way. For MNIST and CIFAR10 such pre-processing has little or no eect. It is also
worth noting that NORB's standard error rate deviation is bigger than CIFAR10's (see Tables 2
and 3). Identical nets with dierent initializations do not produce very consistent results. The
best net had an error rate of 1.72%, the worst 3.69%.
4.3 Experiments on CIFAR 10
CIFAR10 is a collection of natural color images of 32x32 pixels. It contains 10 classes, each of them
with 5000 samples in the training set and 1000 in the test set. The images greatly vary inside each
class. They are not necessarily centered, may contain only parts of the object, and have varying
backgrounds. All of this makes CIFAR10 the hardest problem addressed in this paper. The CNN
has three maps, one for each color channel (RGB). The CIFAR10 images are relatively small in

12 pages, 2 figures, 5 tables


has three maps, one for each color channel (RGB). The CIFAR10 images are relatively small in
comparison to NORB's, and force us to use small kernels. The tested CNNs dier only in the
number of maps per convolutional and max-pooling layer. All have eight hidden layers: layer1, a
convolutional layer with 3 3 kernels and skipping factor of 0; layer2, a max-pooling layer over a
33 region; layer3, a convolutional layer with 3 3 kernels and skipping factors of 0 0; layer4,
a max-pooling over a 2 2 region; layer5, a convolutional layer with 3 3 kernels and a skipping
factors of 0 0; layer6, a max pooling layer over a 2 2 region; layer7, a fully connected layer
with 300 neurons; layer8, a fully connected layer with 100 neurons.
Like for MNIST, the learning rate is initialized by 0.001 and multiplied by 0.993 after every
epoch. Results in Table 3 show that without translation the error rate does not drop below 28%;
adding edge information does not help at all. Translations have a very positive eect, decreasing

12 pages, 2 figures, 5 tables


adding edge information does not help at all. Translations have a very positive eect, decreasing
the error rate to almost 20%. Contrast extraction lters are better than the Sobel/Scharr lters but
still worse than no pre-processing layer at all. Despite some CNN-inherent translation invariance,
additional training image translations cause better generalization; additional image processing
proved useless though.
To see if bigger nets are better, we increase the number of maps per layer from 100 to 200, 300
and 400, respectively (last three rows in Tab. 3). Training time increases exponentially, but the
test error decreases, reaching a minimum for nets with 300 maps per layer. Our 19.51% error rate
is better than the previous state of the art for this dataset, 20.40% (Coates et al. , 2010) and 25.50%
Technical Report No. IDSIA-01-11 9
Table 3: Average error rates and standard deviations for N runs of an eight hidden layer CNN on
the CIFAR10 test set (see text for details). The rst ve nets have 100 maps per convolutional
and max-pooling layer, whereas the sixth, seventh and eighth have 200, 300 and 400 maps per

12 pages, 2 figures, 5 tables


and max-pooling layer, whereas the sixth, seventh and eighth have 200, 300 and 400 maps per
hidden layer, respectively. IP - image processing layer: edge - 3 3 Sobel and Scharr lters; hat 1313 positive and negative contrast extraction lters.
trans. [%] maps IP TfbV [%] runs time/epoch [s]
0 100 no 28.87 0.37 11 93
0 100 edge 29.11 0.36 15 104
5 100 no 20.26 0.21 11 111
5 100 edge 21.87 0.57 5 120
5 100 hat 21.44 0.44 4 136
5 200 no 19.90 0.16 5 248
5 300 no 19.51 0.18 5 532
5 400 no 19.54 0.16 5 875
(Yu and Zhang, 2010). Unlike Coates et al. (2010), however, we use the original images without
any particular input normalization. Note that the error rate standard deviations are smaller than
those obtained on NORB, that is, dierent initializations yield consistent results.
4.4 Speedup factor of GPU code
The GPU code scales well with network size. For small nets the speedup is small (but still over 10)

12 pages, 2 figures, 5 tables


4.4 Speedup factor of GPU code
The GPU code scales well with network size. For small nets the speedup is small (but still over 10)
since they t better inside the CPU cache, and GPU resources are underutilized. For huge nets
(ex: Table 2) the GPU implementation is more than 60 times faster than a compiler-optimized
CPU version. Given the
exibility of our GPU version, this is a signicant speedup. One epoch
takes 35 GPU minutes but more than 35 CPU hours.
5 Conclusion
We presented high-performance GPU-based CNN variants trained by on-line gradient descent, with
sparse random connectivity, computationally more ecient and biologically more plausible than
fully connected CNNs. Principal advantages include state-of-the-art generalization capabilities,
great
exibility and speed. All structural CNN parameters such as input image size, number of
hidden layers, number of maps per layer, kernel sizes, skipping factors and connection tables are
adaptable to any particular application. We applied our networks to benchmark datasets for digit
recognition (MNIST), 3D object recognition (NORB), and natural images (CIFAR10). On MNIST
the best network achieved a recognition test error rate of 0.35%, on NORB 2.53% and on CIFAR10

12 pages, 2 figures, 5 tables


the best network achieved a recognition test error rate of 0.35%, on NORB 2.53% and on CIFAR10
19.51%. Our results are raising the bars for all three benchmarks. Currently the particular CNN
types discussed in this paper seem to be the best adaptive image recognizers, provided there is a
labeled dataset of sucient size. No unsupervised pretraining is required. Good results require
big and deep but sparsely connected CNNs, computationally prohibitive on CPUs, but feasible on
current GPUs, where our implementation is 10 to 60 times faster than a compiler-optimized CPU
version.
Technical Report No. IDSIA-01-11 10
Acknowledgment
This work was partially funded by the Swiss Commission for Technology and Innovation (CTI),
Project n. 9688.1 IFF: Intelligent Fill in Form.
References
S. Behnke. Hierarchical Neural Networks for Image Interpretation , volume 2766 of Lecture Notes
in Computer Science . Springer, 2003.
K. Chellapilla, S. Puri, and P. Simard. High performance convolutional neural networks for
document processing. In International Workshop on Frontiers in Handwriting Recognition ,
2006.
D. C. Cire san, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep big simple neural nets

12 pages, 2 figures, 5 tables


2006.
D. C. Cire san, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep big simple neural nets
for handwritten digit recogntion. Neural Computation , 22(12):3207{3220, 2010.
A. Coates, H. Lee, and A. Ng. An analysis of single-layer networks in unsupervised feature learning.
InAdvances in Neural Information Processing Systems , 2010.
K. Fukushima. Neocognitron: A self-organizing neural network for a mechanism of pattern recognition unaected by shift in position. Biological Cybernetics , 36(4):193{202, 1980.
K. Fukushima. Neocognitron for handwritten digit recognition. Neurocomputing , 51:161{180,
2003.
P. O. Hoyer and A. Hyv arinen. Independent component analysis applied to feature extraction
from colour and stero images. Network: Computation in Neural Systems , 11(3):191{210, 2000.
A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Computer
Science Department, University of Toronto, 2009.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haner. Gradient-based learning applied to document

12 pages, 2 figures, 5 tables


Y. LeCun, L. Bottou, Y. Bengio, and P. Haner. Gradient-based learning applied to document
recognition. Proceedings of the IEEE , 86(11):2278{2324, November 1998.
Y. LeCun, F.-J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proc. of Computer Vision and Pattern Recognition Conference ,
2004.
J. Mutch and D. G. Lowe. Object class recognition and localization using sparse features with
limited receptive elds. Int. J. Comput. Vision , 56(6):503{511, 2008.
V. Nair and G. E. Hinton. 3d object recognition with deep belief nets. In Advances in Neural
Information Processing Systems , 2009.
B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy
employed by v1? Vision Research , 37(23):3311{3325, December 1997.
N. Pinto, D. Doukhan, J. J. DiCarlo, and D. D. Cox. A high-throughput screening approach
to discovering good forms of biologically inspired visual representation. PLoS computational
biology , 5(11):e1000579, November 2009.

12 pages, 2 figures, 5 tables


to discovering good forms of biologically inspired visual representation. PLoS computational
biology , 5(11):e1000579, November 2009.
M. Riesenhuber and T. Poggio. Hierarchical models of object recognition in cortex. Nat. Neurosci. ,
2(11):1019{1025, 1999.
Technical Report No. IDSIA-01-11 11
D. Scherer, A. M uller, and S. Behnke. Evaluation of pooling operations in convolutional architectures for object recognition. In International Conference on Articial Neural Networks , 2010.
J. Schmidhuber, M. Eldracher, and B. Foltin. Semilinear predictability minimization produces
well-known feature detectors. Neural Computation , 8(4):773{786, 1996.
J. Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation ,
4(6):863{879, 1992.
T. Serre, L. Wolf, and T. Poggio. Object recognition with features inspired by visual cortex. In
Proc. of Computer Vision and Pattern Recognition Conference , 2007.
P. Simard, D. Steinkraus, and J. Platt. Best practices for convolutional neural networks applied
to visual document analysis. In Seventh International Conference on Document Analysis and

12 pages, 2 figures, 5 tables


to visual document analysis. In Seventh International Conference on Document Analysis and
Recognition , pages 958{963, 2003.
R. Uetz and S. Behnke. Large-scale object recognition with cuda-accelerated hierarchical neural
networks. In IEEE International Converence on Intelligent Computing and Intelligent Systems
(ICIS) , 2009.
D. H. Wiesel and T. N. Hubel. Receptive elds of single neurones in the cat's striate cortex. J.
Physiol. , 148:574{591, 1959.
K. Yu and T. Zhang. Improved local coordinate coding using local tangents. In Proceedings of the
International Conference on Machine Learning , 2010.

12 pages, 2 figures, 5 tables


Knowledge Matters: Importance of Prior Information for
Optimization
C  a glar G ul cehre gulcehrc@iro.umontreal.ca
D epartement d'informatique et de recherche op erationnelle
Universit e de Montr eal, Montr eal, QC, Canada
Yoshua Bengio bengioy@iro.umontreal.ca
D epartement d'informatique et de recherche op erationnelle
Universit e de Montr eal, Montr eal, QC, Canada
Editor: Not Assigned
Abstract
We explore the eect of introducing prior information into the intermediate level of deep
supervised neural networks for a learning task on which all the black-box state-of-the-art machine learning algorithms tested have failed to learn. We motivate our work from the hypothesis
that there is an optimization obstacle involved in the nature of such tasks, and that humans
learn useful intermediate concepts from other individuals via a form of supervision or guidance
using a curriculum. The experiments we have conducted provide positive evidence in favor of
this hypothesis. In our experiments, a two-tiered MLP architecture is trained on a dataset for

Summary:
We explore the effect of introducing prior information into the intermediate
level of neural networks for a learning task on which all the state-of-the-art
machine learning algorithms tested failed to learn. We motivate our work from
the hypothesis that humans learn such intermediate concepts from other
individuals via a form of supervision or guidance using a curriculum. The
experiments we have conducted provide positive evidence in favor of this
hypothesis. In our experiments, a two-tiered MLP architecture is trained on a
dataset with 64x64 binary inputs images, each image with three sprites. The
final task is to decide whether all the sprites are the same or one of them is
different. Sprites are pentomino tetris shapes and they are placed in an image
with different locations using scaling and rotation transformations. The first
part of the two-tiered MLP is pre-trained with intermediate-level targets being
the presence of sprites at each location, while the second part takes the
output of the first part as input and predicts the final task's target binary
event. The two-tiered MLP architecture, with a few tens of thousand examples,
was able to learn the task perfectly, whereas all other algorithms (include
unsupervised pre-training, but also traditional algorithms like SVMs, decision
trees and boosting) all perform no better than chance. We hypothesize that the
optimization difficulty involved when the intermediate pre-training is not
performed is due to the {\em composition} of two highly non-linear tasks. Our
findings are also consistent with hypotheses on cultural learning inspired by
the observations of optimization problems with deep learning, presumably
because of effective local minima.

37 Pages, 5 figures, 5 tables JMLR Special Topics on Representation
  Learning Submission


using a curriculum. The experiments we have conducted provide positive evidence in favor of
this hypothesis. In our experiments, a two-tiered MLP architecture is trained on a dataset for
which each image input contains three sprites, and the binary target class is 1 if all three have
the same shape. Black-box machine learning algorithms only got chance on this task. Standard
deep supervised neural networks also failed. However, using a particular structure and guiding
the learner by providing intermediate targets in the form of intermediate concepts (the presence of each object) allows to nail the task. Much better than chance but imperfect results are
also obtained by exploring architecture and optimization variants, pointing towards a dicult
optimization task. We hypothesize that the learning diculty is due to the composition of two
highly non-linear tasks. Our ndings are also consistent with hypotheses on cultural learning
inspired by the observations of eective local minima (possibly due to ill-conditioning and the
training procedure not being able to escape what appears like a local minimum).
Keywords: Deep Learning, Neural Networks, Optimization, Evolution of Culture, Curriculum Learning, Training with Hints
1. Introduction
There is a recent emerging interest in dierent elds of science for cultural learning (Henrich
and McElreath, 2003) and how groups of individuals exchanging information can learn in ways


and McElreath, 2003) and how groups of individuals exchanging information can learn in ways
superior to individual learning. This is also witnessed by the emergence of new research elds
such as "Social Neuroscience". Learning from other agents in an environment by the means
of cultural transmission of knowledge with a peer-to-peer communication is an ecient and
natural way of acquiring or propagating common knowledge. The most popular belief on how
the information is transmitted between individuals is that bits of information are transmitted
by small units, called memes, which share some characteristics of genes, such as self-replication,
mutation and response to selective pressures (Dawkins, 1976).
1arXiv:1301.4083v6  [cs.LG]  13 Jul 2013
This paper is based on the hypothesis (which is further elaborated in Bengio (2013a)) that
human culture and the evolution of ideas have been crucial to counter an optimization issue:
this diculty would otherwise make it dicult for human brains to capture high level knowledge of the world without the help of other educated humans. In this paper machine learning
experiments are used to investigate some elements of this hypothesis by seeking answers for
the following questions: are there machine learning tasks which are intrinsically hard for a
lone learning agent but that may become very easy when intermediate concepts are provided


the following questions: are there machine learning tasks which are intrinsically hard for a
lone learning agent but that may become very easy when intermediate concepts are provided
by another agent as additional intermediate learning cues, in the spirit of Curriculum Learning (Bengio et al., 2009b)? What makes such learning tasks more dicult? Can specic initial
values of the neural network parameters yield success when random initialization yield complete failure? Is it possible to verify that the problem being faced is an optimization problem or
with a regularization problem? These are the questions discussed (if not completely addressed)
here, which relate to the following broader question: how can humans (and potentially one day,
machines) learn complex concepts?
In this paper, results of dierent machine learning algorithms on an articial learning task
involving binary 64 64 images are presented. In that task, each image in the dataset contains
3 Pentomino tetris sprites (simple shapes). The task is to gure out if all the sprites in the
image are the same or if there are dierent sprite shapes in the image. Several state-of-the-art
machine learning algorithms have been tested and none of them could perform better than
a random predictor on the test set. Nevertheless by providing hints about the intermediate
concepts (the presence and location of particular sprite classes), the problem can easily be solved


a random predictor on the test set. Nevertheless by providing hints about the intermediate
concepts (the presence and location of particular sprite classes), the problem can easily be solved
where the same-architecture neural network without the intermediate concepts guidance fails.
Surprisingly, our attempts at solving this problem with unsupervised pre-training algorithms
failed solve this problem. However, with specic variations in the network architecture or
training procedure, it is found that one can make a big dent in the problem. For showing the
impact of intermediate level guidance, we experimented with a two-tiered neural network, with
supervised pre-training of the rst part to recognize the category of sprites independently of
their orientation and scale, at dierent locations, while the second part learns from the output
of the rst part and predicts the binary task of interest.
The objective of this paper is not to propose a novel learning algorithm or architecture,
but rather to rene our understanding of the learning diculties involved with composed tasks
(here a logical formula composed with the detection of object classes), in particular the training
diculties involved for deep neural networks. The results also bring empirical evidence in favor
of some of the hypotheses from Bengio (2013a), discussed below, as well as introducing a
particular form of curriculum learning (Bengio et al., 2009b).


of some of the hypotheses from Bengio (2013a), discussed below, as well as introducing a
particular form of curriculum learning (Bengio et al., 2009b).
Building dicult AI problems has a long history in computer science. Specically hard
AI problems have been studied to create CAPTCHA's that are easy to solve for humans, but
hard to solve for machines (Von Ahn et al., 2003). In this paper we are investigating a dicult
problem for the o-the-shelf black-box machine learning algorithms.1
1.1 Curriculum Learning and Cultural Evolution Against Eective Local Minima
What Bengio (2013a) calls an eective local minimum is a point where iterative training
stalls, either because of an actual local minimum or because the optimization algorithm is
1. You can access the source code of some experiments presented in that paper and their hyperparameters from
here: https://github.com/caglar/kmatters
2
unable (in reasonable time) to nd a descent path (e.g., because of serious ill-conditioning). In
this paper, it is hypothesized that some more abstract learning tasks such as those obtained
by composing simpler tasks are more likely to yield eective local minima for neural networks,
and are generally hard for general-purpose machine learning algorithms.


by composing simpler tasks are more likely to yield eective local minima for neural networks,
and are generally hard for general-purpose machine learning algorithms.
The idea that learning can be enhanced by guiding the learner through intermediate easier
tasks is old, starting with animal training by shaping (Skinner, 1958; Peterson, 2004; Krueger
and Dayan, 2009). Bengio et al. (2009b) introduce a computational hypothesis related to a
presumed issue with eective local minima when directly learning the target task: the good
solutions correspond to hard-to-nd-by-chance eective local minima, and intermediate tasks
prepare the learner's internal conguration (parameters) in a way similar to continuation methods in global optimization (which go through a sequence of intermediate optimization problems,
starting with a convex one where local minima are no issue, and gradually morphing into the
target task of interest).
In a related vein, Bengio (2013a) makes the following inferences based on experimental
observations of deep learning and neural network learning:
Point 1: Training deep architectures is easier when some hints are given about the function
that the intermediate levels should compute (Hinton et al., 2006; Weston et al., 2008;
Salakhutdinov and Hinton, 2009; Bengio, 2009). The experiments performed here expand


Salakhutdinov and Hinton, 2009; Bengio, 2009). The experiments performed here expand
in particular on this point.
Point 2: It is much easier to train a neural network with supervision (where examples ar
provided to it of when a concept is present and when it is not present in a variety of
examples) than to expect unsupervised learning to discover the concept (which may also
happen but usually leads to poorer renditions of the concept). The poor results obtained
with unsupervised pre-training reinforce that hypothesis .
Point 3: Directly training all the layers of a deep network together not only makes it dicult to
exploit all the extra modeling power of a deeper architecture but in many cases it actually
yields worse results as the number of required layers is increased (Larochelle et al., 2009;
Erhan et al., 2010). The experiments performed here also reinforce that hypothesis.
Point 4: Erhan et al. (2010) observed that no two training trajectories ended up in the
same eective local minimum, out of hundreds of runs, even when comparing solutions as
functions from input to output, rather than in parameter space (thus eliminating from the
picture the presence of symmetries and multiple local minima due to relabeling and other


functions from input to output, rather than in parameter space (thus eliminating from the
picture the presence of symmetries and multiple local minima due to relabeling and other
reparametrizations). This suggests that the number of dierent eective local minima
(even when considering them only in function space) must be huge.
Point 5: Unsupervised pre-training, which changes the initial conditions of the descent procedure, sometimes allows to reach substantially better eective local minima (in terms
of generalization error!), and these better local minima do not appear to be reachable
by chance alone (Erhan et al., 2010). The experiments performed here provide another
piece of evidence in favor of the hypothesis that where random initialization can yield
rather poor results, specically targeted initialization can have a drastic impact, i.e., that
3
eective local minima are not just numerous but that some small subset of them are much
better and hard to reach by chance.2
Based on the above points, Bengio (2013a) then proposed the following hypotheses regarding
learning of high-level abstractions.
Optimization Hypothesis: When it learns, a biological agent performs an approximate
optimization with respect to some implicit objective function.


learning of high-level abstractions.
Optimization Hypothesis: When it learns, a biological agent performs an approximate
optimization with respect to some implicit objective function.
Deep Abstractions Hypothesis: Higher level abstractions represented in brains require deeper computations (involving the composition of more non-linearities).
Local Descent Hypothesis: The brain of a biological agent relies on approximate local
descent and gradually improves itself while learning.
Eective Local Minima Hypothesis: The learning process of a single human learner
(not helped by others) is limited by eective local minima.
Deeper Harder Hypothesis: Eective local minima are more likely to hamper learning
as the required depth of the architecture increases.
Abstractions Harder Hypothesis: High-level abstractions are unlikely to be discovered by a single human learner by chance, because these abstractions are represented by
a deep subnetwork of the brain, which learns by local descent.
Guided Learning Hypothesis: A human brain can learn high level abstractions if
guided by the signals produced by other agents that act as hints or indirect supervision
for these high-level abstractions.
Memes Divide-and-Conquer Hypothesis: Linguistic exchange, individual learning


for these high-level abstractions.
Memes Divide-and-Conquer Hypothesis: Linguistic exchange, individual learning
and the recombination of memes constitute an ecient evolutionary recombination operator in the meme-space. This helps human learners to collectively build better internal
representations of their environment, including fairly high-level abstractions.
This paper is focused on \ Point 1 " and testing the \ Guided Learning Hypothesis ", using
machine learning algorithms to provide experimental evidence. The experiments performed
also provide evidence in favor of the \ Deeper Harder Hypothesis " and associated \ Abstractions
Harder Hypothesis ". Machine Learning is still far beyond the current capabilities of humans,
and it is important to tackle the remaining obstacles to approach AI. For this purpose, the
question to be answered is why tasks that humans learn eortlessly from very few examples,
while machine learning algorithms fail miserably?
2. Recent work showed that rather deep feedforward networks can be very successfully trained when large
quantities of labeled data are available (Ciresan et al., 2010; Glorot et al., 2011a; Krizhevsky et al., 2012).
Nonetheless, the experiments reported here suggest that it all depends on the task being considered, since
even with very large quantities of labeled examples, the deep networks trained here were unsuccessful.
4


Nonetheless, the experiments reported here suggest that it all depends on the task being considered, since
even with very large quantities of labeled examples, the deep networks trained here were unsuccessful.
4
2. Culture and Optimization Diculty
As hypothesized in the \ Local Descent Hypothesis ", human brains would rely on a local approximate descent, just like a Multi-Layer Perceptron trained by a gradient-based iterative optimization. The main argument in favor of this hypothesis relies on the biologically-grounded
assumption that although ring patterns in the brain change rapidly, synaptic strengths underlying these neural activities change only gradually, making sure that behaviors are generally
consistent across time. If a learning algorithm is based on a form of local (e.g. gradient-based)
descent, it can be sensitive to eective local minima (Bengio, 2013a).
When one trains a neural network, at some point in the training phase the evaluation of
error seems to saturate, even if new examples are introduced. In particular Erhan et al. (2010)
nd that early examples have a much larger weight in the nal solution. It looks like the learner
is stuck in or near a local minimum. But since it is dicult to verify if this is near a true local


is stuck in or near a local minimum. But since it is dicult to verify if this is near a true local
minimum or simply an eect of strong ill-conditioning, we call such a \stuck" conguration an
eective local minimum , whose denition depends not just on the optimization objective but
also on the limitations of the optimization algorithm.
Erhan et al. (2010) highlighted both the issue of eective local minima and a regularization eect when initializing a deep network with unsupervised pre-training. Interestingly, as
the network gets deeper the diculty due to eective local minima seems to be get more pronounced. That might be because of the number of eective local minima increases (more like
an actual local minima issue), or maybe because the good ones are harder to reach (more like
an ill-conditioning issue) and more work will be needed to clarify this question.
As a result of Point 4 we hypothesize that it is very dicult for an individual's brain to
discover some higher level abstractions by chance only. As mentioned in the \ Guided Learning
Hypothesis " humans get hints from other humans and learn high-level concepts with the guidance of other humans3. Curriculum learning (Bengio et al., 2009a) and incremental learning


(Solomono, 1989), are examples of this. This is done by properly choosing the sequence of
examples seen by the learner, where simpler examples are introduced rst and more complex
examples shown when the learner is ready for them. One of the hypothesis on why curriculum
works states that curriculum learning acts as a continuation method that allows one to discover
a good minimum, by rst nding a good minimum of a smoother error function. Recent experiments on human subjects also indicates that humans teach by using a curriculum strategy
(Khan et al., 2011).
Some parts of the human brain are known to have a hierarchical organization (i.e. visual
cortex) consistent with the deep architecture studied in machine learning papers. As we go from
the sensory level to higher levels of the visual cortex, we nd higher level areas corresponding
to more abstract concepts. This is consistent with the Deep Abstractions Hypothesis .
Training neural networks and machine learning algorithms by decomposing the learning
task into sub-tasks and exploiting prior information about the task is well-established and
in fact constitutes the main approach to solving industrial problems with machine learning.
The contribution of this paper is rather on rendering explicit the eective local minima issue
and providing evidence on the type of problems for which this diculty arises. This prior


The contribution of this paper is rather on rendering explicit the eective local minima issue
and providing evidence on the type of problems for which this diculty arises. This prior
information and hints given to the learner can be viewed as inductive bias for a particular task,
an important ingredient to obtain a good generalization error (Mitchell, 1980). An interesting
3. But some high-level concepts may also be hardwired in the brain, as assumed in the universal grammar
hypothesis (Montague, 1970), or in nature vs nurture discussions in cognitive science.
5
earlier nding in that line of research was done with Explanation Based Neural Networks
(EBNN) in which a neural network transfers knowledge across multiple learning tasks. An
EBNN uses previously learned domain knowledge as an initialization or search bias (i.e. to
constrain the learner in the parameter space) (O'Sullivan, 1996; Mitchell and Thrun, 1993).
Another related work in machine learning is mainly focused on reinforcement learning algorithms, based on incorporating prior knowledge in terms of logical rules to the learning
algorithm as a prior knowledge to speed up and bias learning (Kunapuli et al., 2010; Towell
and Shavlik, 1994).
As discussed in \ Memes Divide and Conquer Hypothesis \ societies can be viewed as a


and Shavlik, 1994).
As discussed in \ Memes Divide and Conquer Hypothesis \ societies can be viewed as a
distributed computational processing systems. In civilized societies knowledge is distributed
across dierent individuals, this yields a space eciency. Moreover computation, i.e. each
individual can specialize on a particular task/topic, is also divided across the individuals in the
society and hence this will yield a computational eciency. Considering the limitations of the
human brain, the whole processing can not be done just by a single agent in an ecient manner.
A recent study in paleoantropology states that there is a substantial decline in endocranial
volume of the brain in the last 30000 years Henneberg (1988). The volume of the brain shrunk
to 1241 ml from 1502 ml (Henneberg and Steyn, 1993). One of the hypothesis on the reduction
of the volume of skull claims that, decline in the volume of the brain might be related to the
functional changes in brain that arose as a result of cultural development and emergence of
societies given that this time period overlaps with the transition from hunter-gatherer lifestyle
to agricultural societies.
3. Experimental Setup
Some tasks, which seem reasonably easy for humans to learn4, are nonetheless appearing almost
impossible to learn for current generic state-of-art machine learning algorithms.


3. Experimental Setup
Some tasks, which seem reasonably easy for humans to learn4, are nonetheless appearing almost
impossible to learn for current generic state-of-art machine learning algorithms.
Here we study more closely such a task, which becomes learnable if one provides hints to
the learner about appropriate intermediate concepts. Interestingly, the task we used in our
experiments is not only hard for deep neural networks but also for non-parametric machine
learning algorithms such as SVM's, boosting and decision trees.
The result of the experiments for varying size of dataset with several o-the-shelf black box
machine learning algorithms and some popular deep learning algorithms are provided in Table
1. The detailed explanations about the algorithms and the hyperparameters used for those
algorithms are given in the Appendix Section 5.2. We also provide some explanations about
the methodologies conducted for the experiments at Section 3.2.
3.1 Pentomino Dataset
In order to test our hypothesis, an articial dataset for object recognition using 64 64 binary
images is designed5. If the task is two tiered (i.e., with guidance provided), the task in the
rst part is to recognize and locate each Pentomino object class6in the image. The second
4. keeping in mind that humans can exploit prior knowledge, either from previous learning or innate knowledge.


4. keeping in mind that humans can exploit prior knowledge, either from previous learning or innate knowledge.
5. The source code for the script that generates the articial Pentomino datasets (Arcade-Universe) is available
at:https://github.com/caglar/Arcade-Universe . This implementation is based on Olivier Breuleux's
bugland dataset generator.
6. A human learner does not seem to need to be taught the shape categories of each Pentomino sprite in order
to solve the task. On the other hand, humans have lots of previously learned knowledge about the notion of
shape and how central it is in dening categories.
6
(a) sprites, not all same type
 (b) sprites, all of same type
Figure 1: Left (a): An example image from the dataset which has a dierent sprite type in it.
Right (b): An example image from the dataset that has only one type of Pentomino
object in it, but with dierent orientations and scales.
part/nal binary classication task is to gure out if all the Pentominos in the image are of
the same shape class or not. If a neural network learned to detect the categories of each object
at each location in an image, the remaining task becomes an XOR-like operation between the


the same shape class or not. If a neural network learned to detect the categories of each object
at each location in an image, the remaining task becomes an XOR-like operation between the
detected object categories. The types of Pentomino objects that is used for generating the
dataset are as follows:
Pentomino sprites N, P, F, Y, J, and Q, along with the Pentomino N2 sprite (mirror of
\Pentomino N" sprite), the Pentomino F2 sprite (mirror of \Pentomino F" sprite), and the
Pentomino Y2 sprite (mirror of \Pentomino Y" sprite).
Figure 2: Dierent classes of Pentomino shapes used in our dataset.
As shown in Figures 1(a) and 1(b), the synthesized images are fairly simple and do not
have any texture. Foreground pixels are \1" and background pixels are \0". Images of the
training and test sets are generated iid. For notational convenience, assume that the domain
of raw input images is X, the set of sprites is S, the set of intermediate object categories is Y
for each possible location in the image and the set of nal binary task outcomes is Z=f0;1g.


for each possible location in the image and the set of nal binary task outcomes is Z=f0;1g.
Two dierent types of rigid body transformation is performed: sprite rotation rot(X;
) where
 =f
: (
= 90)^[(2N);(03)]gand scaling scale (X;) where2f1;2gis the
scaling factor. The data generating procedure is summarized below.
Sprite transformations: Before placing the sprites in an empty image, for each image x2X,
a value for z2Zis randomly sampled which is to have (or not) the same three sprite
shapes in the image. Conditioned on the constraint given by z, three sprites are randomly
7
selectedsijfromSwithout replacement. Using a uniform probability distribution over
all possible scales, a scale is chosen and accordingly each sprite image is scaled. Then
rotate each sprite is randomly rotated by a multiple of 90 degrees.
Sprite placement: Upon completion of sprite transformations, a 64 64 uniform grid is generated which is divided into 8 8 blocks, each block being of size 8 8 pixels, and randomly
select three dierent blocks from the 64=8 8 on the grid and place the transformed


select three dierent blocks from the 64=8 8 on the grid and place the transformed
objects into dierent blocks (so they cannot overlap, by construction).
Each sprite is centered in the block in which it is located. Thus there is no object translation
inside the blocks. The only translation invariance is due to the location of the block inside the
image.
A Pentomino sprite is guaranteed to not over
ow the block in which it is located, and there
are no collisions or overlaps between sprites, making the task simpler. The largest possible
Pentomino sprite can be t into an 8 4 mask.
3.2 Learning Algorithms Evaluated
Initially the models are cross-validated by using 5-fold cross-validation. With 40,000 examples,
this gives 32,000 examples for training and 8,000 examples for testing. For neural network
algorithms, stochastic gradient descent (SGD) is used for training. The following standard
learning algorithms were rst evaluated: decision trees, SVMs with Gaussian kernel, ordinary
fully-connected Multi-Layer Perceptrons, Random Forests, k-Nearest Neighbors, Convolutional
Neural Networks, and Stacked Denoising Auto-Encoders with supervised ne-tuning. More


Neural Networks, and Stacked Denoising Auto-Encoders with supervised ne-tuning. More
details of the congurations and hyper-parameters for each of them are given in Appendix
Section 5.2. The only better than chance results were obtained with variations of the Structured
Multi-Layer Perceptron described below.
3.2.1 Structured Multi-Layer Perceptron (SMLP)
The neural network architecture that is used to solve this task is called the SMLP (Structured
Multi-Layer Perceptron), a deep neural network with two parts as illustrated in Figure 5 and
7:
The lower part, P1NN ( Part 1 Neural Network , as it is called in the rest of the paper),
has shared weights and local connectivity, with one identical MLP instance of the P1NN for
each patch of the image, and typically an 11-element output vector per patch (unless otherwise
noted). The idea is that these 11 outputs per patch could represent the detection of the sprite
shape category (or the absence of sprite in the patch). The upper part, P2NN ( Part 2 Neural
Network ) is a fully connected one hidden layer MLP that takes the concatenation of the outputs


Network ) is a fully connected one hidden layer MLP that takes the concatenation of the outputs
of all patch-wise P1NNs as input. Note that the rst layer of P1NN is similar to a convolutional
layer but where the stride equals the kernel size, so that windows do not overlap, i.e., P1NN can
be decomposed into separate networks sharing the same parameters but applied on dierent
patches of the input image, so that each network can actually be trained patch-wise in the case
where a target is provided for the P1NN outputs. The P1NN output for patch piwhich is
extracted from the image xis computed as follows:
f(pi) =g2(Vg1(Upi+b) +c) (1)
8
where pi2Rdis the input patch/receptive eld extracted from location iof a single image.
U2Rdhdis the weight matrix for the rst layer of P1NN and b2Rd
his the vector of biases
for the rst layer of P1NN. g1() is the activation function of the rst layer and g2() is the
activation function of the second layer. In many of the experiments, best results were obtained


activation function of the second layer. In many of the experiments, best results were obtained
withg1() a rectifying non-linearity (a.k.a. as RELU), which is max(0; X) (Jarrett et al.,
2009b; Nair and Hinton, 2010; Glorot et al., 2011a; Krizhevsky et al., 2012). V2Rdhdois the
second layer's weights matrix, such that and c2Rdoare the biases of the second layer of the
P1NN, with doexpected to be smaller than dh.
In this way, g1(Upi+b) is an overcomplete representation of the input patch that can
potentially represent all the possible Pentomino shapes for all factors of variations in the patch
(rotation, scaling and Pentomino shape type). On the other hand, when trained with hints,
f(pi) is expected to be the lower dimensional representation of a Pentomino shape category
invariant to scaling and rotation in the given patch.
In the experiments with SMLP trained with hints (targets at the output of P1NN), the
P1NN is expected to perform classication of each 8 8 non-overlapping patches of the original


P1NN is expected to perform classication of each 8 8 non-overlapping patches of the original
6464 input image without having any prior knowledge of whether that specic patch contains
a Pentomino shape or not. P1NN in SMLP without hints just outputs the local activations for
each patch, and gradients on f(pi) are backpropagated from the upper layers. In both cases
P1NN produces the input representation for the Part 2 Neural Net (P2NN). Thus the input
representation of P2NN is the concatenated output of P1NN across all the 64 patch locations:
ho= [f(p0);:::;f(pi);:::;f(pN))] whereNis the number of patches and the ho2Rdi;di=
doN.hois the concatenated output of the P1NN at each patch.
There is a standardization layer on top of the output of P1NN that centers the activations
and performs divisive normalization by dividing by the standard deviation over a minibatch
of the activations of that layer. We denote the standardization function z(). Standardization
makes use of the mean and standard deviation computed for each hidden unit such that each


of the activations of that layer. We denote the standardization function z(). Standardization
makes use of the mean and standard deviation computed for each hidden unit such that each
hidden unit of howill have 0 activation and unit standard deviation on average over the
minibatch. Xis the set of pentomino images in the minibatch, where X2RdinNis a matrix
withNimages.h(i)
o(xj) is the vector of activations of the i-th hidden unit of hidden layer
ho(xj) for thej-th example, with xj2X.
h(i)
o=1
NX
xj2Xh(i)
o(xj) (2)
h(i)
o=sPN
j(h(i)
o(xj)h(i)
o)2
N+ (3)
z(h(i)
o(xj)) =h(i)
o(xj)h(i)
o
max(h(i)
o;)(4)
whereis a very small constant, that is used to prevent numerical under
ows in the standard
deviation. P1NN is trained on each 8 8 patches extracted from the image.


deviation. P1NN is trained on each 8 8 patches extracted from the image.
hois standardized for each training and test sample separately. Dierent values of were
used for SMLP-hints and SMLP-nohints.
The concatenated output of P1NN is fed as an input to the P2NN. P2NN is a feedforward
MLP with a sigmoid output layer using a single RELU hidden layer. The task of P2NN is to
perform a nonlinear logical operation on the representation provided at the output of P1NN.
9
3.2.2 Structured Multi Layer Perceptron Trained with Hints (SMLP-hints)
The SMLP-hints architecture exploits a hint about the presence and category of Pentomino
objects, specifying a semantics for the P1NN outputs. P1NN is trained with the intermediate
targetY, specifying the type of Pentomino sprite shape present (if any) at each of the 64 patches
(88 non-overlapping blocks) of the image. Because a possible answer at a given location can
be \none of the object types" i.e., an empty patch, yp(for patchp) can take one of the 11


be \none of the object types" i.e., an empty patch, yp(for patchp) can take one of the 11
possible values, 1 for rejection and the rest is for the Pentomino shape classes, illustrated in
Figure 2:
yp=(
0 if patch pis empty
s2Sif the patch pcontains a Pentomino sprite :
A similar task has been studied by Fleuret et al. (2011) (at SI appendix Problem 17), who
compared the performance of humans vs computers.
The SMLP-hints architecture takes advantage of dividing the task into two subtasks during
training with prior information about intermediate-level relevant factors. Because the sum of
the training losses decomposes into the loss on each patch, the P1NN can be pre-trained patchwise. Each patch-specic component of the P1NN is a fully connected MLP with 8 8 inputs
and 11 outputs with a softmax output layer. SMLP-hints uses the the standardization given
in Equation 3 but with = 0.
The standardization is a crucial step for training the SMLP on the Pentomino dataset, and
yields much sparser outputs, as seen on Figures 3 and 4. If the standardization is not used,


yields much sparser outputs, as seen on Figures 3 and 4. If the standardization is not used,
even SMLP-hints could not solve the Pentomino task. In general, the standardization step
dampens the small activations and augments larger ones(reducing the noise). Centering the
activations of each feature detector in a neural network has been studied in (Raiko et al., 2012)
and (Vatanen et al., 2013). They proposed that transforming the outputs of each hidden neuron
in a multi-layer perceptron network to have zero output and zero slope on average makes rst
order optimization methods closer to the second order techniques.
By default, the SMLP uses rectier hidden units as activation function, we found a signicant boost by using rectication compared to hyperbolic tangent and sigmoid activation
functions. The P1NN has a highly overcomplete architecture with 1024 hidden units per patch,
and L1 and L2 weight decay regularization coecients on the weights (not the biases) are respectively 1e-6 and 1e-5. The learning rate for the P1NN is 0.75. 1 training epoch was enough


for the P1NN to learn the features of Pentomino shapes perfectly on the 40000 training examples. The P2NN has 2048 hidden units. L1 and L2 penalty coecients for the P2NN are
1e-6, and the learning rate is 0.1. These were selected by trial and error based on validation
set error. Both P1NN (for each patch) and P2NN are fully-connected neural networks, even
though P1NN globally is a special kind of convolutional neural network.
Filters of the rst layer of SMLP are shown in Figure 6. These are the examples of the
lters obtained with the SLMP-hints trained with 40k examples, whose results are given in
Table 1. Those lters look very noisy but they work perfectly on the Pentomino task.
3.2.3 Deep and Structured Supervised MLP without Hints (SMLP-nohints)
SMLP-nohints uses the same connectivity pattern (and deep architecture) that is also used
in the SMLP-hints architecture, but without using the intermediate targets ( Y). It directly
predicts the nal outcome of the task ( Z), using the same number of hidden units, the same
10


predicts the nal outcome of the task ( Z), using the same number of hidden units, the same
10
Figure 3: Bar chart of concatenated softmax output activations hoof P1NN (1164=704 outputs) in SMLP-hints before standardization, for a selected example. There are very
large spikes at each location for one of the possible 11 outcome (1 of K representation).
11
Figure 4: Softmax output activations hoof P1NN at SMLP-hints before standardization. There
are positive spiked outputs at the locations where there is a Pentomino shape. Positive and negative spikes arise because most of the outputs are near an average value.
Activations are higher at the locations where there is a pentomino shape.
12
Structured MLP
Architecture with
Hints
Final Binary task
labels
Intermediate level
targets .
Second
Level
Neural
Network
First
Level
Neural
Network
Figure 5: Structured MLP architecture, used with hints (trained in two phases, rst P1NN,
bottom two layers, then P2NN, top two layers). In SMLP-hints, P1NN is trained


bottom two layers, then P2NN, top two layers). In SMLP-hints, P1NN is trained
on each 8x8 patch extracted from the image and the softmax output probabilities of
all 64 patches are concatenated into a 64 11 vector that forms the input of P2NN.
OnlyUandVare learned in the P1NN and its output on each patch is fed into
P2NN. The rst level and the second level neural networks are trained separately,
not jointly.
Figure 6: Filters of Structured MLP architecture, trained with hints on 40k examples.
13
connectivity and the same activation function for the hidden units as SMLP-hints. 120 hyperparameter values have been evaluated by randomly selecting the number of hidden units
from [64;128;256;512;1024;1200;2048] and randomly sampling 20 learning rates uniformly in
the log-domain within the interval of [0 :008;0:8]. Two fully connected hidden layers with 1024
hidden units (same as P1NN) per patch is used and 2048 (same as P2NN) for the last hidden layer, with twenty training epochs. For this network the best results are obtained with a
learning rate of 0.05.7
Structured MLP
Architecture without
Hints
Final Binary task


learning rate of 0.05.7
Structured MLP
Architecture without
Hints
Final Binary task
labels
Second
Level
Neural
Network
First
Level
Neural
Network
Figure 7: Structured MLP architecture, used without hints (SMLP-nohints). It is the same
architecture as SMLP-hints (Figure 5) but with both parts (P1NN and P2NN) trained
jointly with respect to the nal binary classication task.
We chose to experiment with various SMLP-nohint architectures and optimization procedures, trying unsuccessfully to achieve as good results with SMLP-nohint as with SMLP-hints.
Rectier Non-Linearity A rectier nonlinearity is used for the activations of MLP hidden
layers. We observed that using piecewise linear nonlinearity activation function such as the
rectier can make the optimization more tractable.
7. The source code of the structured MLP is available at the github repository: https://github.com/caglar/
structured_mlp
14


7. The source code of the structured MLP is available at the github repository: https://github.com/caglar/
structured_mlp
14
Figure 8: First layer lters learned by the Structured MLP architecture, trained without using hints on 447600 examples with online SGD and a sigmoid intermediate layer
activation.
Intermediate Layer The output of the P1NN is considered as an intermediate layer of the
SMLP. For the SMLP-hints, only softmax output activations have been tried at the intermediate
layer, and that suced to learn the task. Since things did not work nearly as well with
the SMLP-nohints, several dierent activation functions have been tried: softmax( ), tanh(),
sigmoid() and linear activation functions.
Standardization Layer Normalization at the last layer of the convolutional neural networks
has been used occasionaly to encourage the competition between the hidden units. (Jarrett
et al., 2009a) used a local contrast normalization layer in their architecture which performs
subtractive and divisive normalization. A local contrast normalization layer enforces a local
competition between adjacent features in the feature map and between features at the same


subtractive and divisive normalization. A local contrast normalization layer enforces a local
competition between adjacent features in the feature map and between features at the same
spatial location in dierent feature maps. Similarly (Krizhevsky et al., 2012) observed that
using a local response layer that enjoys the benet of using local normalization scheme aids
generalization.
Standardization has been observed to be crucial for both SMLP trained with or without hints. In both SMLP-hints and SMLP-nohints experiments, the neural network was not
able to generalize or even learn the training set without using standardization in the SMLP
intermediate layer, doing just chance performance. More specically, in the SMLP-nohints architecture, standardization is part of the computational graph, hence the gradients are being
backpropagated through it. The mean and the standard deviation is computed for each hidden
unit separately at the intermediate layer as in Equation 4. But in order to prevent numerical
under
ows or over
ows during the backpropagation we have used = 1e8 (Equation 3).
The benet of having sparse activations may be specically important for the ill-conditioned


The benet of having sparse activations may be specically important for the ill-conditioned
problems, for the following reasons. When a hidden unit is \o", its gradient (the derivative of
the loss with respect to its output) is usually close to 0 as well, as seen here. That means that all
o-diagonal second derivatives involving that hidden unit (e.g. its input weights) are also near 0.
This is basically like removing some columns and rows from the Hessian matrix associated with
a particular example. It has been observed that the condition number of the Hessian matrix
(specically, its largest eigenvalue) increases as the size of the network increases (Dauphin and
Bengio, 2013), making training considerably slower and inecient (Dauphin and Bengio, 2013).
Hence one would expect that as sparsity of the gradients (obtained because of sparsity of the
activations) increases, training would become more ecient, as if we were training a smaller
sub-network for each example, with shared weights across examples, as in dropouts (Hinton
et al., 2012).
In Figure 9, the activation of each hidden unit in a bar chart is shown: the eect of
standardization is signicant, making the activations sparser.


In Figure 9, the activation of each hidden unit in a bar chart is shown: the eect of
standardization is signicant, making the activations sparser.
15
(a) Before standardization.
 (b) After standardization.
Figure 9: Activations of the intermediate-level hidden units of an SLMP-nohints for a particular
examples (x-axis: hidden unit number, y-axis: activation value). Left (a): before
standardization. Right (b): after standardization.
In Figure 10, one can see the activation histogram of the SMLP-nohints intermediate layer,
showing the distribution of activation values, before and after standardization. Again the
sparsifying eect of standardization is very apparent.
(a) Before standardization.
 (b) After standardization.
Figure 10: Distribution histogram of activation values of SMLP-nohints intermediate layer.
Left (a): before standardization. Right (b): after standardization.
In Figures 10 and 9, the intermediate level activations of SMLP-nohints are shown before
and after standardization. These are for the same SMLP-nohints architecture whose results
are presented on Table 1. For that same SMLP, the Adadelta (Zeiler, 2012) adaptive learning
16


are presented on Table 1. For that same SMLP, the Adadelta (Zeiler, 2012) adaptive learning
16
rate scheme has been used, with 512 hidden units for the hidden layer of P1NN and rectier
activation function. For the output of the P1NN, 11 sigmoidal units have been used while
P2NN had 1200 hidden units with rectier activation function. The output nonlinearity of the
P2NN is a sigmoid and the training objective is the binary crossentropy.
Adaptive Learning Rates We have experimented with several dierent adaptive learning
rate algorithms. We tried rmsprop8, Adadelta (Zeiler, 2012), Adagrad (Duchi et al., 2010) and
a linearly (1/t) decaying learning rate (Bengio, 2013b). For the SMLP-nohints with sigmoid
activation function we have found Adadelta(Zeiler, 2012) converging faster to an eective local
minima and usually yielding better generalization error compared to the others.
3.2.4 Deep and Structured MLP with Unsupervised Pre-Training
Several experiments have been conducted using an architecture similar to the SMLP-nohints,


3.2.4 Deep and Structured MLP with Unsupervised Pre-Training
Several experiments have been conducted using an architecture similar to the SMLP-nohints,
but by using unsupervised pre-training of P1NN, with Denoising Auto-Encoder (DAE) and/or
Contractive Auto-Encoders (CAE). Supervised ne-tuning proceeds as in the deep and structured MLP without hints. Because an unsupervised learner may not focus the representation
just on the shapes, a larger number of intermediate-level units at the output of P1NN has been
explored: previous work on unsupervised pre-training generally found that larger hidden layers
were optimal when using unsupervised pre-training, because not all unsupervised features will
be relevant to the task at hand. Instead of limiting to 11 units per patch, we experimented
with networks with up to 20 hidden (i.e., code) units per patch in the second-layer patch-wise
auto-encoder.
In Appendix 5.1 we also provided the result of some experiments with binary-binary RBMs
trained on 88 patches from the 40k training dataset.
In unsupervised pretraining experiments in this paper, both contractive auto-encoder (CAE)


trained on 88 patches from the 40k training dataset.
In unsupervised pretraining experiments in this paper, both contractive auto-encoder (CAE)
with sigmoid nonlinearity and binary cross entropy cost function and denoising auto-encoder (DAE)
have been used. In the second layer, experiments were performed with a DAE with rectier
hidden units utilizing L1 sparsity and weight decay on the weights of the auto-encoder. Greedy
layerwise unsupervised training procedure is used to train the deep auto-encoder architecture
(Bengio et al., 2007). In unsupervised pretraining experiments, tied weights have been used.
Dierent combinations of CAE and DAE for unsupervised pretraining have been tested, but
none of the congurations tested managed to learn the Pentomino task, as shown in Table 1.
3.3 Experiments with 1 of K representation
To explore the eect of changing the complexity of the input representation on the diculty
of the task, a set of experiments have been designed with symbolic representations of the
information in each patch. In all cases an empty patch is represented with a 0 vector. These
representation can be seen as an alternative input for a P2NN-like network, i.e., they were fed


representation can be seen as an alternative input for a P2NN-like network, i.e., they were fed
as input to an MLP or another black-box classier.
The following four experiments have been conducted, each one using one using a dierent
input representation for each patch:
8. This is learning rate scaling method that is discussed by G. Hinton in his Video Lecture 6.5 - rmsprop: Divide
the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine
Learning, 2012.
17
Algorithm 20k dataset 40k dataset 80k dataset
Training Test Training Test Training Test
Error Error Error Error Error Error
SVM RBF 26.2 50.2 28.2 50.2 30.2 49.6
K Nearest Neighbors 24.7 50.0 25.3 49.5 25.6 49.0
Decision Tree 5.8 48.6 6.3 49.4 6.9 49.9
Randomized Trees 3.2 49.8 3.4 50.5 3.5 49.1
MLP 26.5 49.3 33.2 49.9 27.2 50.1
Convnet/Lenet5 50.6 49.8 49.4 49.8 50.2 49.8


Convnet/Lenet5 50.6 49.8 49.4 49.8 50.2 49.8
Maxout Convnet 14.5 49.5 0.0 50.1 0.0 44.6
2 layer sDA 49.4 50.3 50.2 50.3 49.7 50.3
Struct. Supervised MLP w/o hints 0.0 48.6 0.0 36.0 0.0 12.4
Struct. MLP+CAE Supervised Finetuning 50.5 49.7 49.8 49.7 50.3 49.7
Struct. MLP+CAE+DAE, Supervised Finetuning 49.1 49.7 49.4 49.7 50.1 49.7
Struct. MLP+DAE+DAE, Supervised Finetuning 49.5 50.3 49.7 49.8 50.3 49.7
Struct. MLP with Hints 0.21 30.7 0 3.1 0 0.01
Table 1: The error percentages with dierent learning algorithms on Pentomino dataset with
dierent number of training examples.


Table 1: The error percentages with dierent learning algorithms on Pentomino dataset with
dierent number of training examples.
Experiment 1-Onehot representation without transformations: In this experiment several trials have been done with a 10-input one-hot vector per patch. Each input corresponds to an object category given in clear, i.e., the ideal input for P2NN if a supervised
P1NN perfectly did its job.
Experiment 2-Disentangled representations: In this experiment, we did trials with 16
binary inputs per patch, 10 one-hot bits for representing each object category, 4 for
rotations and 2 for scaling, i.e., the whole information about the input is given, but it is
perfectly disentangled. This would be the ideal input for P2NN if an unsupervised P1NN
perfectly did its job.
Experiment 3-Onehot representation with transformations: For each of the ten object
types there are 8 = 4 2 possible transformations. Two objects in two dierent patches
are the considered \the same" (for the nal task) if their category is the same regardless
of the transformations. The one-hot representation of a patch corresponds to the crossproduct between the 10 object shape classes and the 4 2 transformations, i.e., one out of


of the transformations. The one-hot representation of a patch corresponds to the crossproduct between the 10 object shape classes and the 4 2 transformations, i.e., one out of
80=1042 possibilities represented in an 80-bit one-hot vector. This also contains all
the information about the input image patch, but spread out in a kind of non-parametric
and non-informative (not disentangled) way, like a perfect memory-based unsupervised
learner (like clustering) could produce. Nevertheless, the shape class would be easier to
read out from this representation than from the image representation (it would be an OR
over 8 of the bits).
Experiment 4-Onehot representation with 80 choices: This representation has the same
1 of 80 one-hot representation per patch but the target task is dened dierently. Two objects in two dierent patches are considered the same i they have exactly the same 80-bit
onehot representation (i.e., are of the same object category with the same transformation
applied).
The rst experiment is a sanity check. It was conducted with single hidden-layered MLP's
with rectier and tanh nonlinearity, and the task was learned perfectly (0 error on both training
and test dataset) with very few training epochs.
18


with rectier and tanh nonlinearity, and the task was learned perfectly (0 error on both training
and test dataset) with very few training epochs.
18
0 100 200 300 400 500 600 700 800 9000.00.10.20.30.40.50.6
Training Error Rate
Test Error Rate(a) Training and Test Errors for Experiment 4
0 100 200 300 400 500 600 700 8000.00.10.20.30.40.50.6
Training Error Rate
Test Error Rate (b) Training and Test Errors for Experiment 3
Figure 11: Tanh MLP training curves. Left (a): The training and test errors of Experiment
3 over 800 training epochs with 100k training examples using Tanh MLP. Right
(b):The training and test errors of Experiment 4 over 700 training epochs with 100k
training examples using Tanh MLP.
The results of Experiment 2 are given in Table 2. To improve results, we experimented
with the Maxout non-linearity in a feedforward MLP (Goodfellow et al., 2013) with two hidden
layers. Unlike the typical Maxout network mentioned in the original paper, regularizers have
been deliberately avoided in order to focus on the optimization issue, i.e: no weight decay, norm
constraint on the weights, or dropout. Although learning from a disentangled representation is


constraint on the weights, or dropout. Although learning from a disentangled representation is
more dicult than learning from perfect object detectors, it is feasible with some architectures
such as the Maxout network. Note that this representation is the kind of representation that
one could hope an unsupervised learning algorithm could discover, at best, as argued in Bengio
et al. (2012).
The only results obtained on the validation set for Experiment 3 and Experiment 4 are
shown respectively in Table 3 and Table 4. In these experiments a tanh MLP with two hidden
layers have been tested with the same hyperparameters. In experiment 3 the complexity of the
problem comes from the transformations (8=4 2) and the number of object types.
But in experiment 4, the only source of complexity of the task comes from the number of
dierent object types. These results are in between the complete failure and complete success
observed with other experiments, suggesting that the task could become solvable with better
training or more training examples. Figure 11 illustrates the progress of training a tanh MLP,
on both the training and test error, for Experiments 3 and 4. Clearly, something has been
learned, but the task is not nailed yet. On experiment 3 for both maxout and tanh the maxout
there was a long plateau where the training error and objective stays almost same. Maxout


there was a long plateau where the training error and objective stays almost same. Maxout
did just chance on the experiment for about 120 iterations on the training and the test set.
But after 120th iteration the training and test error started decline and eventually it was able
to solve the task. Moreover as seen from the curves in Figure 11(a) and 11(b), the training
and test error curves are almost the same for both tasks. This implies that for onehot inputs,
whether you increase the number of possible transformations for each object or the number of
19
Learning Algorithm Training Error Test Error
SVM 0.0 35.6
Random Forests 1.29 40.475
Tanh MLP 0.0 0.0
Maxout MLP 0.0 0.0
Table 2: Performance of dierent learning algorithms on disentangled representation in Experiment 2.
Learning Algorithm Training Error Test Error
SVM 11.212 32.37
Random Forests 24.839 48.915
Tanh MLP 0.0 22.475
Maxout MLP 0.0 0.0
Table 3: Performance of dierent learning algorithms using a dataset with onehot vector and
80 inputs as discussed for Experiment 3.
object categories, as soon as the number of possible congurations is same, the complexity of
the problem is almost the same for the MLP.


object categories, as soon as the number of possible congurations is same, the complexity of
the problem is almost the same for the MLP.
3.4 Does the Eect Persist with Larger Training Set Sizes?
The results shown in this section indicate that the problem in the Pentomino task clearly
is not just a regularization problem, but rather basically hinges on an optimization problem.
Otherwise, we would expect test error to decrease as the number of training examples increases.
This is shown rst by studying the online case and then by studying the ordinary training
case with a xed size training set but considering increasing training set sizes. In the online
minibatch setting, parameter updates are performed as follows:
t+1=tt (5)
t=PN
irtL(xt;t)
N(6)
whereL(xt;t) is the loss incurred on example xtwith parameters t, wheret2Z+and
is the learning rate.
Ordinary batch algorithms converge linearly to the optimum , however the noisy gradient
estimates in the online SGD will cause parameter to
uctuate near the local optima. However,
online SGD directly optimizes the expected risk, because the examples are drawn iid from the


online SGD directly optimizes the expected risk, because the examples are drawn iid from the
ground-truth distribution (Bottou, 2010). Thus:
L1=E[L(x;)] =Z
xL(x;)p(x)dx (7)
20
Learning Algorithm Training Error Test Error
SVM 4.346 40.545
Random Forests 23.456 47.345
Tanh MLP 0 25.8
Table 4: Performance of dierent algorithms using a dataset with onehot vector and 80 binary
inputs as discussed in Experiment 4.
whereL1is the generalization error. Therefore online SGD is trying to minimize the
expected risk with noisy updates. Those noisy updates have the eect of regularizer:
t=PN
irtL(xt;t)
N=rtL(x;t) +t (8)
wherertL(x;t) is the true gradient and tis the zero-mean stochastic gradient \noise"
due to computing the gradient over a nite-size minibatch sample.
We would like to know if the problem with the Pentomino dataset is more a regularization


due to computing the gradient over a nite-size minibatch sample.
We would like to know if the problem with the Pentomino dataset is more a regularization
or an optimization problem. An SMLP-nohints model was trained by online SGD with the randomly generated online Pentomino stream. The learning rate was adaptive, with the Adadelta
procedure (Zeiler, 2012) on minibatches of 100 examples. In the online SGD experiments, two
SMLP-nohints that is trained with and without standardization at the intermediate layer with
exactly the same hyperparameters are tested. The SMLP-nohints P1NN patch-wise submodel
has 2048 hidden units and the SMLP intermediate layer has 1152 = 64 18 hidden units. The
nonlinearity that is used for the intermediate layer is the sigmoid. P2NN has 2048 hidden units.
SMLP-nohints has been trained either with or without standardization on top of the output
units of the P1NN. The experiments illustrated in Figures 12 and 13 are with the same SMLP
without hints architecture for which results are given in Table 1. In those graphs only the results
for the training on the randomly generated 545400 Pentomino samples have been presented. As


without hints architecture for which results are given in Table 1. In those graphs only the results
for the training on the randomly generated 545400 Pentomino samples have been presented. As
shown in the plots SMLP-nohints was not able to generalize without standardization. Although
without standardization the training loss seems to decrease initially, it eventually gets stuck in
a plateau where training loss doesn't change much.
Training of SMLP-nohints online minibatch SGD is performed using standardization in the
intermediate layer and Adadelta learning rate adaptation, on 1046000 training examples from
the randomly generated Pentomino stream. At the end of the training, test error is down to
27.5%, which is much better than chance but from from the score obtained with SMLP-hints
of near 0 error.
In another SMLP-nohints experiment without standardization the model is trained with the
1580000 Pentomino examples using online minibatch SGD. P1NN has 2048 hidden units and 16
sigmoidal outputs per patch. for the P1NN hidden layer. P2NN has 1024 hidden units for the
hidden layer. Adadelta is used to adapt the learning rate. At the end of training this SMLP,
the test error remained stuck, at 50.1%.
21
0 100 200 300 400 500 600 700 800


the test error remained stuck, at 50.1%.
21
0 100 200 300 400 500 600 700 800
Batch no0.200.250.300.350.400.450.500.55Test Error
SMLP with standardization
SMLP without standardizationFigure 12: Test errors of SMLP-nohints with and without standardization in the intermediate
layer. Sigmoid as an intermediate layer activation has been used. Each tick (batch
no) in the x-axis represents 400 examples.
22
0 100 200 300 400 500 600 700
Batch no01234567Training LossSMLP with standardization
SMLP without standardizationFigure 13: Training errors of SMLP-nohints with and without standardization in the intermediate layer. Sigmoid nonlinearity has been used as an intermediate layer activation
function. The x-axis is in units of blocks of 400 examples in the training set.
23
3.4.1 Experiments with Increased Training Set Size
Here we consider the eect of training dierent learners with dierent numbers of training
examples. For the experimental results shown in Table 1, 3 training set sizes (20k, 40k and 80k
examples) had been used. Each dataset was generated with dierent random seeds (so they


examples) had been used. Each dataset was generated with dierent random seeds (so they
do not overlap). Figure 14 also shows the error bars for an ordinary MLP with three hidden
layers, for a larger range of training set sizes, between 40k and 320k examples. The number of
training epochs is 8 (more did not help), and there are three hidden layers with 2048 feature
detectors. The learning rate we used in our experiments is 0.01. The activation function of the
MLP is a tanh nonlinearity, while the L1, L2 penalty coecients are both 1e-6.
Table 1 shows that, without guiding hints, none of the state-of-art learning algorithms could
perform noticeably better than a random predictor on the test set. This shows the importance
of intermediate hints introduced in the SMLP. The decision trees and SVMs can overt the
training set but they could not generalize on the test set. Note that the numbers reported in the
table are for hyper-parameters selected based on validation set error, hence lower training errors
are possible if avoiding all regularization and taking large enough models. On the training set,
the MLP with two large hidden layers (several thousands) could reach nearly 0% training error,
but still did not manage to achieve good test error.