A LOCALISATION
In the main bodyof the paper we have consideredthe classiﬁca tion task of the ILSVRC challenge,
and performed a thorough evaluation of ConvNet architectur es of different depth. In this section,
we turn to the localisation task of the challenge, which we ha ve won in 2014 with 25.3%error. It
can be seen as a special case of object detection, where a sing le object bounding box should be
predictedforeach ofthe top-5classes, irrespectiveof the actual numberofobjectsof the class. For
thiswe adoptthe approachof Sermanetet al. (2014), the winn ersof the ILSVRC-2013localisation
challenge,withafewmodiﬁcations. Ourmethodisdescribed inSect.A.1andevaluatedinSect.A.2.
A.1 L OCALISATION CONVNET
To perform object localisation, we use a very deep ConvNet, w here the last fully connected layer
predicts the bounding box location instead of the class scor es. A bounding box is represented by
a 4-D vector storing its center coordinates, width, and heig ht. There is a choice of whether the


a 4-D vector storing its center coordinates, width, and heig ht. There is a choice of whether the
boundingbox prediction is shared across all classes (singl e-class regression, SCR (Sermanetet al.,
2014))orisclass-speciﬁc(per-classregression,PCR).In theformercase,thelastlayeris4-D,while
in the latter it is 4000-D (since there are 1000 classes in the dataset). Apart from the last bounding
boxpredictionlayer,weuse theConvNetarchitectureD (Tab le1),whichcontains16weightlayers
andwasfoundtobe thebest-performingin theclassiﬁcation task (Sect.4).
Training. Training of localisation ConvNets is similar to that of the c lassiﬁcation ConvNets
(Sect.3.1). Themaindifferenceisthatwereplacethelogis ticregressionobjectivewithaEuclidean
loss,whichpenalisesthedeviationofthepredictedboundi ngboxparametersfromtheground-truth.
We trainedtwo localisation models, each on a single scale: S= 256andS= 384(due to the time

Very Deep Convolutional Networks for Large-Scale Image Recognition

Summary:
In this work we investigate the effect of the convolutional network depth on
its accuracy in the large-scale image recognition setting. Our main
contribution is a thorough evaluation of networks of increasing depth using an
architecture with very small (3x3) convolution filters, which shows that a
significant improvement on the prior-art configurations can be achieved by
pushing the depth to 16-19 weight layers. These findings were the basis of our
ImageNet Challenge 2014 submission, where our team secured the first and the
second places in the localisation and classification tracks respectively. We
also show that our representations generalise well to other datasets, where
they achieve state-of-the-art results. We have made our two best-performing
ConvNet models publicly available to facilitate further research on the use of
deep visual representations in computer vision.

Material Recognition in the Wild with the Materials in Context Database

Bell, S., Upchurch, P., Snavely, N., Bala, K.

Return of the Devil in the Details: Delving Deep into Convolutional Nets

Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.

Deep Convolutional Filter Banks for Texture Recognition and Segmentation

Flexible, High Performance Convolutional Neural Networks for Image Classification

Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., Schmidhuber, J.


We trainedtwo localisation models, each on a single scale: S= 256andS= 384(due to the time
constraints,we didnot use trainingscale jitteringforour ILSVRC-2014submission). Trainingwas
initialised with the correspondingclassiﬁcation models ( trained on the same scales), and the initial
learning rate was set to 10−3. We exploredboth ﬁne-tuningall layers and ﬁne-tuningonly the ﬁrst
two fully-connected layers, as done in (Sermanetetal., 201 4). The last fully-connected layer was
initialisedrandomlyandtrainedfromscratch.
Testing. We consider two testing protocols. The ﬁrst is used for compa ring different network
modiﬁcations on the validation set, and considers only the b oundingbox prediction for the ground
truth class (to factor out the classiﬁcation errors). The bo unding box is obtained by applying the
networkonlyto thecentralcropoftheimage.
The second, fully-ﬂedged, testing procedure is based on the dense application of the localisation
ConvNet to the whole image, similarly to the classiﬁcation t ask (Sect. 3.2). The difference is that


ConvNet to the whole image, similarly to the classiﬁcation t ask (Sect. 3.2). The difference is that
instead of the class score map, the output of the last fully-c onnected layer is a set of bounding
box predictions. To come up with the ﬁnal prediction, we util ise the greedy merging procedure
of Sermanetetal. (2014), which ﬁrst merges spatially close predictions (by averaging their coordinates), and then rates them based on the class scores, obta ined from the classiﬁcation ConvNet.
When several localisation ConvNets are used, we ﬁrst take th e union of their sets of boundingbox
predictions, and then run the mergingprocedureon the union . We did not use the multiple pooling
10
Publishedasa conferencepaperat ICLR2015
offsets technique of Sermanetetal. (2014), which increase s the spatial resolution of the bounding
boxpredictionsandcanfurtherimprovetheresults.
A.2 L OCALISATION EXPERIMENTS


boxpredictionsandcanfurtherimprovetheresults.
A.2 L OCALISATION EXPERIMENTS
In this section we ﬁrst determine the best-performinglocal isation setting (using the ﬁrst test protocol), and then evaluate it in a fully-ﬂedged scenario (the se cond protocol). The localisation error
is measured according to the ILSVRC criterion (Russakovsky et al., 2014), i.e. the bounding box
predictionis deemed correctif its intersectionoverunion ratio with the ground-truthboundingbox
isabove0.5.
Settings comparison. As can be seen from Table 8, per-class regression (PCR) outpe rforms the
class-agnostic single-class regression (SCR), which diff ers from the ﬁndings of Sermanetetal.
(2014), where PCR was outperformed by SCR. We also note that ﬁ ne-tuning all layers for the localisation task leads to noticeablybetter results than ﬁne -tuningonly the fully-connectedlayers(as
donein(Sermanetet al.,2014)). Intheseexperiments,thes mallestimagessidewassetto S= 384;
theresultswith S= 256exhibitthesamebehaviourandarenotshownforbrevity.


theresultswith S= 256exhibitthesamebehaviourandarenotshownforbrevity.
Table 8:Localisation error for different modiﬁcations with the simpliﬁed testing protocol: the
boundingbox is predictedfrom a single central image crop, a nd the ground-truthclass is used. All
ConvNet layers (except for the last one) have the conﬁgurati on D (Table 1), while the last layer
performseithersingle-classregression(SCR) orper-clas sregression(PCR).
Fine-tunedlayers regression type GTclass localisationerror
1st and2nd FCSCR 36.4
PCR 34.3
all PCR 33.1
Fully-ﬂedgedevaluation. Havingdeterminedthebestlocalisationsetting(PCR,ﬁne- tuningofall
layers),we nowapply it in the fully-ﬂedgedscenario,where the top-5class labelsare predictedusing our best-performingclassiﬁcation system (Sect. 4.5), and multiple densely-computedbounding


box predictions are merged using the method of Sermanetetal . (2014). As can be seen from Table 9, applicationof the localisationConvNetto the whole i magesubstantiallyimprovesthe results
compared to using a center crop (Table 8), despite using the t op-5 predicted class labels instead of
thegroundtruth. Similarlytotheclassiﬁcationtask(Sect .4),testingatseveralscalesandcombining
thepredictionsofmultiplenetworksfurtherimprovesthep erformance.
Table9:Localisationerror
smallestimage side top-5localisationerror (%)
train(S) test(Q) val. test.
256 256 29.5 384 384 28.2 26.7
384 352,384 27.5 fusion: 256/256 and 384/352,384 26.9 25.3
Comparison with the state of the art. We compare our best localisation result with the state
of the art in Table 10. With 25.3%test error, our “VGG” team won the localisation challenge of
ILSVRC-2014 (Russakovskyet al., 2014). Notably, our resul ts are considerably better than those
of the ILSVRC-2013winnerOverfeat(Sermanetet al., 2014), even thoughwe used less scales and


of the ILSVRC-2013winnerOverfeat(Sermanetet al., 2014), even thoughwe used less scales and
did not employ their resolution enhancement technique. We e nvisage that better localisation performance can be achieved if this technique is incorporated i nto our method. This indicates the
performanceadvancementbroughtbyourverydeepConvNets– wegotbetterresultswithasimpler
localisationmethod,buta morepowerfulrepresentation.
B GENERALISATION OF VERYDEEPFEATURES
In the previous sections we have discussed training and eval uation of very deep ConvNets on the
ILSVRC dataset. In this section, we evaluate our ConvNets, p re-trained on ILSVRC, as feature
11
Publishedasa conferencepaperat ICLR2015
Table 10: Comparison with the state of the art in ILSVRC localisation . Our methodis denoted
as“VGG”.
Method top-5val. error (%) top-5 testerror (%)
VGG 26.9 25.3
GoogLeNet (Szegedyet al., 2014) - 26.7
OverFeat (Sermanet etal.,2014) 30.0 29.9
Krizhevsky et al.(Krizhevsky et al.,2012) - 34.2


Krizhevsky et al.(Krizhevsky et al.,2012) - 34.2
extractors on other, smaller, datasets, where training lar ge models from scratch is not feasible due
to over-ﬁtting. Recently, there has been a lot of interest in such a use case (Zeiler&Fergus, 2013;
Donahueet al., 2013; Razavianet al., 2014; Chatﬁeldet al., 2014), as it turns out that deep image
representations,learntonILSVRC,generalisewelltoothe rdatasets,wheretheyhaveoutperformed
hand-crafted representations by a large margin. Following that line of work, we investigate if our
modelsleadtobetterperformancethanmoreshallowmodelsu tilisedinthestate-of-the-artmethods.
In this evaluation, we consider two models with the best clas siﬁcation performance on ILSVRC
(Sect.4)–conﬁgurations“Net-D”and“Net-E”(whichwemade publiclyavailable).
To utilise the ConvNets, pre-trained on ILSVRC, for image cl assiﬁcation on other datasets, we


To utilise the ConvNets, pre-trained on ILSVRC, for image cl assiﬁcation on other datasets, we
remove the last fully-connected layer (which performs 1000 -way ILSVRC classiﬁcation), and use
4096-Dactivationsofthepenultimatelayerasimagefeatur es,whichareaggregatedacrossmultiple
locations and scales. The resulting image descriptor is L2-normalised and combined with a linear
SVM classiﬁer, trained on the target dataset. For simplicit y, pre-trained ConvNet weights are kept
ﬁxed(noﬁne-tuningisperformed).
Aggregation of features is carried out in a similar manner to our ILSVRC evaluation procedure
(Sect. 3.2). Namely, an image is ﬁrst rescaled so that its sma llest side equals Q, and then the network is densely applied over the image plane (which is possib le when all weight layers are treated
as convolutional). We then perform global average pooling o n the resulting feature map, which
producesa 4096-Dimage descriptor. The descriptor is then a veraged with the descriptor of a horizontally ﬂipped image. As was shown in Sect. 4.2, evaluation over multiple scales is beneﬁcial, so


we extract features over several scales Q. The resulting multi-scale features can be either stacked
or pooled across scales. Stacking allows a subsequent class iﬁer to learn how to optimally combine
image statistics over a range of scales; this, however, come s at the cost of the increased descriptor
dimensionality. We returntothediscussionofthisdesignc hoicein theexperimentsbelow. We also
assess late fusion of features, computed using two networks , which is performed by stacking their
respectiveimagedescriptors.
Table11: Comparisonwiththestateoftheartinimageclassiﬁcationo nVOC-2007,VOC-2012,
Caltech-101, and Caltech-256 . Our models are denoted as “VGG”. Results marked with * were
achievedusingConvNetspre-trainedonthe extended ILSVRCdataset(2000classes).
MethodVOC-2007 VOC-2012 Caltech-101 Caltech-256
(meanAP) (mean AP) (meanclass recall) (mean class recall)
Zeiler& Fergus (Zeiler&Fergus, 2013) - 79.0 86.5±0.5 74.2±0.3


Zeiler& Fergus (Zeiler&Fergus, 2013) - 79.0 86.5±0.5 74.2±0.3
Chatﬁeldetal. (Chatﬁeldet al., 2014) 82.4 83.2 88.4±0.6 77.6±0.1
He etal. (Heet al.,2014) 82.4 - 93.4±0.5 Weiet al.(Weiet al., 2014) 81.5(85.2∗)81.7 (90.3∗) - VGGNet-D (16layers) 89.3 89.0 91.8±1.0 85.0±0.2
VGGNet-E(19 layers) 89.3 89.0 92.3±0.5 85.1±0.3
VGGNet-D & Net-E 89.7 89.3 92.7±0.5 86.2±0.3
Image Classiﬁcation on VOC-2007and VOC-2012. We beginwith the evaluationon the image
classiﬁcation task of PASCAL VOC-2007 and VOC-2012 benchma rks (Everinghametal., 2015).


classiﬁcation task of PASCAL VOC-2007 and VOC-2012 benchma rks (Everinghametal., 2015).
These datasets contain 10K and 22.5K images respectively, a nd each image is annotated with one
or several labels, correspondingto 20 object categories. T he VOC organisersprovidea pre-deﬁned
split into training, validation, and test data (the test dat a for VOC-2012 is not publicly available;
instead,anofﬁcialevaluationserverisprovided). Recogn itionperformanceismeasuredusingmean
averageprecision(mAP)acrossclasses.
Notably, by examining the performance on the validation set s of VOC-2007 and VOC-2012, we
foundthat aggregatingimage descriptors,computedat mult iple scales, by averagingperformssim12
Publishedasa conferencepaperat ICLR2015
ilarly to the aggregation by stacking. We hypothesize that t his is due to the fact that in the VOC


Publishedasa conferencepaperat ICLR2015
ilarly to the aggregation by stacking. We hypothesize that t his is due to the fact that in the VOC
dataset the objects appear over a variety of scales, so there is no particular scale-speciﬁc semantics which a classiﬁer could exploit. Since averaging has a b eneﬁt of not inﬂating the descriptor dimensionality, we were able to aggregated image descri ptors over a wide range of scales:
Q∈ {256,384,512,640,768}. It is worth noting though that the improvement over a smalle r
rangeof{256,384,512}wasrathermarginal( 0.3%).
Thetestsetperformanceisreportedandcomparedwithother approachesinTable11. Ournetworks
“Net-D”and“Net-E”exhibitidenticalperformanceonVOCda tasets,andtheircombinationslightly
improves the results. Our methods set the new state of the art across image representations, pretrained on the ILSVRC dataset, outperformingthe previousb est result of Chatﬁeldet al. (2014) by
more than 6%. It should be noted that the method of Wei et al. (2014), which achieves1%better


more than 6%. It should be noted that the method of Wei et al. (2014), which achieves1%better
mAP on VOC-2012, is pre-trained on an extended 2000-class IL SVRC dataset, which includes
additional 1000 categories, semantically close to those in VOC datasets. It also beneﬁts from the
fusionwith anobjectdetection-assistedclassiﬁcation pi peline.
ImageClassiﬁcationonCaltech-101andCaltech-256. InthissectionweevaluateverydeepfeaturesonCaltech-101(Fei-Feiet al.,2004)andCaltech-256 (Grifﬁnet al.,2007)imageclassiﬁcation
benchmarks. Caltech-101contains9Kimageslabelledinto1 02classes(101objectcategoriesanda
backgroundclass), while Caltech-256 is larger with 31K ima ges and 257 classes. A standard evaluation protocolon these datasets is to generateseveral ran domsplits into training and test data and
report the average recognition performance across the spli ts, which is measured by the mean class
recall(whichcompensatesforadifferentnumberoftestima gesperclass). FollowingChatﬁeld etal.


recall(whichcompensatesforadifferentnumberoftestima gesperclass). FollowingChatﬁeld etal.
(2014); Zeiler&Fergus(2013); He etal. (2014),onCaltech- 101we generated3 randomsplits into
training and test data, so that each split contains 30 traini ng images per class, and up to 50 test
images per class. On Caltech-256 we also generated 3 splits, each of which contains 60 training
images per class (and the rest is used for testing). In each sp lit, 20% of training images were used
asa validationset forhyper-parameterselection.
We found that unlike VOC, on Caltech datasets the stacking of descriptors, computed over multiple scales, performs better than averaging or max-pooling. This can be explained by the fact that
in Caltech images objects typically occupy the whole image, so multi-scale image features are semanticallydifferent(capturingthe wholeobject vs. object parts), andstacking allows a classiﬁer to
exploitsuchscale-speciﬁcrepresentations. We usedthree scalesQ∈ {256,384,512}.
Ourmodelsarecomparedtoeachotherandthestateofthearti nTable11. Ascanbeseen,thedeeper


Ourmodelsarecomparedtoeachotherandthestateofthearti nTable11. Ascanbeseen,thedeeper
19-layerNet-Eperformsbetterthanthe16-layerNet-D,and theircombinationfurtherimprovesthe
performance. On Caltech-101, our representations are comp etitive with the approach of He etal.
(2014),which,however,performssigniﬁcantlyworsethano urnetsonVOC-2007. OnCaltech-256,
ourfeaturesoutperformthestate oftheart (Chatﬁeldetal. , 2014)byalargemargin( 8.6%).
Action Classiﬁcation on VOC-2012. We also evaluated our best-performing image representation (the stacking of Net-D and Net-E features) on the PASCAL VOC-2012 action classiﬁcation
task (Everinghamet al., 2015), which consists in predictin g an action class from a single image,
given a bounding box of the person performing the action. The dataset contains 4.6K training images,labelledinto11classes. SimilarlytotheVOC-2012ob jectclassiﬁcationtask,theperformance


is measured using the mAP. We considered two training settin gs: (i) computing the ConvNet featuresonthewholeimageandignoringtheprovidedboundingb ox;(ii)computingthefeaturesonthe
wholeimageandontheprovidedboundingbox,andstackingth emtoobtaintheﬁnalrepresentation.
TheresultsarecomparedtootherapproachesinTable12.
OurrepresentationachievesthestateofartontheVOCactio nclassiﬁcationtaskevenwithoutusing
the provided bounding boxes, and the results are further imp roved when using both images and
bounding boxes. Unlike other approaches, we did not incorpo rate any task-speciﬁc heuristics, but
reliedontherepresentationpowerofverydeepconvolution alfeatures.
Other Recognition Tasks. Since the public release of our models, they have been active ly used
by the research community for a wide range of image recogniti on tasks, consistently outperforming more shallow representations. For instance, Girshicke t al. (2014) achieve the state of the
object detection results by replacing the ConvNet of Krizhe vskyet al. (2012) with our 16-layer


object detection results by replacing the ConvNet of Krizhe vskyet al. (2012) with our 16-layer
model. Similar gains over a more shallow architecture of Kri zhevskyet al. (2012) have been ob13
Publishedasa conferencepaperat ICLR2015
Table 12: Comparison with the state of the art in single-image action c lassiﬁcation on VOC2012. Our models are denoted as “VGG”. Results marked with * were a chieved using ConvNets
pre-trainedonthe extended ILSVRCdataset (1512classes).
Method VOC-2012 (meanAP)
(Oquab et al., 2014) 70.2∗
(Gkioxari etal.,2014) 73.6
(Hoai,2014) 76.3
VGG Net-D& Net-E,image-only 79.2
VGG Net-D& Net-E,image and bounding box 84.0
served in semantic segmentation (Longet al., 2014), image c aption generation (Kirosetal., 2014;
Karpathy& Fei-Fei, 2014),textureandmaterialrecognitio n(Cimpoiet al., 2014; Bell etal., 2014).
C PAPERREVISIONS


C PAPERREVISIONS
Here we present the list of major paper revisions, outlining the substantial changes for the convenienceofthe reader.
v1Initialversion. Presentstheexperimentscarriedoutbefo rethe ILSVRCsubmission.
v2Addspost-submissionILSVRCexperimentswithtrainingset augmentationusingscalejittering,
whichimprovestheperformance.
v3Addsgeneralisationexperiments(AppendixB) on PASCAL VOC andCaltech image classiﬁcationdatasets. Themodelsusedforthese experimentsarepub liclyavailable.
v4The paper is converted to ICLR-2015 submission format. Also adds experiments with multiple
cropsforclassiﬁcation.
v6Camera-readyICLR-2015conferencepaper. Addsa compariso nof the net B with a shallow net
andtheresultsonPASCAL VOCactionclassiﬁcationbenchmar k.
14


Fully Convolutional Networks for Semantic Segmentation
Jonathan LongEvan ShelhamerTrevor Darrell
UC Berkeley
fjonlong,shelhamer,trevor g@cs.berkeley.edu
Abstract
Convolutional networks are powerful visual models that
yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixelsto-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build “fully convolutional”
networks that take input of arbitrary size and produce
correspondingly-sized output with efﬁcient inference and
learning. We deﬁne and detail the space of fully convolutional networks, explain their application to spatially dense
prediction tasks, and draw connections to prior models. We
adapt contemporary classiﬁcation networks (AlexNet [19],
the VGG net [31], and GoogLeNet [32]) into fully convolutional networks and transfer their learned representations
by ﬁne-tuning [4] to the segmentation task. We then deﬁne a novel architecture that combines semantic information from a deep, coarse layer with appearance information
from a shallow, ﬁne layer to produce accurate and detailed

Summary:
Convolutional networks are powerful visual models that yield hierarchies of
features. We show that convolutional networks by themselves, trained
end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic
segmentation. Our key insight is to build "fully convolutional" networks that
take input of arbitrary size and produce correspondingly-sized output with
efficient inference and learning. We define and detail the space of fully
convolutional networks, explain their application to spatially dense prediction
tasks, and draw connections to prior models. We adapt contemporary
classification networks (AlexNet, the VGG net, and GoogLeNet) into fully
convolutional networks and transfer their learned representations by
fine-tuning to the segmentation task. We then define a novel architecture that
combines semantic information from a deep, coarse layer with appearance
information from a shallow, fine layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves state-of-the-art
segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012),
NYUDv2, and SIFT Flow, while inference takes one third of a second for a
typical image.

Jonathan Long
Evan Shelhamer
Trevor Darrell

to appear in CVPR (2015)


from a shallow, ﬁne layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves
state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2,
and SIFT Flow, while inference takes less than one ﬁfth of a
second for a typical image.
1. Introduction
Convolutional networks are driving advances in recognition. Convnets are not only improving for whole-image
classiﬁcation [19, 31, 32], but also making progress on local tasks with structured output. These include advances in
bounding box object detection [29, 12, 17], part and keypoint prediction [39, 24], and local correspondence [24, 9].
The natural next step in the progression from coarse to
ﬁne inference is to make a prediction at every pixel. Prior
approaches have used convnets for semantic segmentation
[27, 2, 8, 28, 16, 14, 11], in which each pixel is labeled with
the class of its enclosing object or region, but with shortcomings that this work addresses.
Authors contributed equally
963842564096409621
21backward/learningforward/inference
pixelwise predictionsegmentation g.t.

to appear in CVPR (2015)


Authors contributed equally
963842564096409621
21backward/learningforward/inference
pixelwise predictionsegmentation g.t.
256384Figure 1. Fully convolutional networks can efﬁciently learn to
make dense predictions for per-pixel tasks like semantic segmentation.
We show that a fully convolutional network (FCN),
trained end-to-end, pixels-to-pixels on semantic segmentation exceeds the state-of-the-art without further machinery. To our knowledge, this is the ﬁrst work to train FCNs
end-to-end (1) for pixelwise prediction and (2) from supervised pre-training. Fully convolutional versions of existing
networks predict dense outputs from arbitrary-sized inputs.
Both learning and inference are performed whole-image-ata-time by dense feedforward computation and backpropagation. In-network upsampling layers enable pixelwise prediction and learning in nets with subsampled pooling.
This method is efﬁcient, both asymptotically and absolutely, and precludes the need for the complications in other
works. Patchwise training is common [27, 2, 8, 28, 11], but

to appear in CVPR (2015)


works. Patchwise training is common [27, 2, 8, 28, 11], but
lacks the efﬁciency of fully convolutional training. Our approach does not make use of pre- and post-processing complications, including superpixels [8, 16], proposals [16, 14],
or post-hoc reﬁnement by random ﬁelds or local classiﬁers
[8, 16]. Our model transfers recent success in classiﬁcation [19, 31, 32] to dense prediction by reinterpreting classiﬁcation nets as fully convolutional and ﬁne-tuning from
their learned representations. In contrast, previous works
have applied small convnets without supervised pre-training
[8, 28, 27].
Semantic segmentation faces an inherent tension between semantics and location: global information resolves
what while local information resolves where. Deep feature
1arXiv:1411.4038v2  [cs.CV]  8 Mar 2015
hierarchies jointly encode location and semantics in a localto-global pyramid. We deﬁne a novel “skip” architecture
to combine deep, coarse, semantic information and shallow,
ﬁne, appearance information in Section 4.2 (see Figure 3).

to appear in CVPR (2015)


to combine deep, coarse, semantic information and shallow,
ﬁne, appearance information in Section 4.2 (see Figure 3).
In the next section, we review related work on deep classiﬁcation nets, FCNs, and recent approaches to semantic
segmentation using convnets. The following sections explain FCN design and dense prediction tradeoffs, introduce
our architecture with in-network upsampling and multilayer combinations, and describe our experimental framework. Finally, we demonstrate state-of-the-art results on
PASCAL VOC 2011-2, NYUDv2, and SIFT Flow.
2. Related work
Our approach draws on recent successes of deep nets
for image classiﬁcation [19, 31, 32] and transfer learning
[4, 38]. Transfer was ﬁrst demonstrated on various visual
recognition tasks [4, 38], then on detection, and on both
instance and semantic segmentation in hybrid proposalclassiﬁer models [12, 16, 14]. We now re-architect and ﬁnetune classiﬁcation nets to direct, dense prediction of semantic segmentation. We chart the space of FCNs and situate
prior models, both historical and recent, in this framework.
Fully convolutional networks To our knowledge, the

to appear in CVPR (2015)


prior models, both historical and recent, in this framework.
Fully convolutional networks To our knowledge, the
idea of extending a convnet to arbitrary-sized inputs ﬁrst
appeared in Matan et al. [25], which extended the classic
LeNet [21] to recognize strings of digits. Because their net
was limited to one-dimensional input strings, Matan et al.
used Viterbi decoding to obtain their outputs. Wolf and Platt
[37] expand convnet outputs to 2-dimensional maps of detection scores for the four corners of postal address blocks.
Both of these historical works do inference and learning
fully convolutionally for detection. Ning et al. [27] deﬁne
a convnet for coarse multiclass segmentation of C. elegans
tissues with fully convolutional inference.
Fully convolutional computation has also been exploited
in the present era of many-layered nets. Sliding window
detection by Sermanet et al . [29], semantic segmentation
by Pinheiro and Collobert [28], and image restoration by
Eigen et al. [5] do fully convolutional inference. Fully convolutional training is rare, but used effectively by Tompson
et al. [35] to learn an end-to-end part detector and spatial

to appear in CVPR (2015)


et al. [35] to learn an end-to-end part detector and spatial
model for pose estimation, although they do not exposit on
or analyze this method.
Alternatively, He et al . [17] discard the nonconvolutional portion of classiﬁcation nets to make a
feature extractor. They combine proposals and spatial
pyramid pooling to yield a localized, ﬁxed-length feature
for classiﬁcation. While fast and effective, this hybrid
model cannot be learned end-to-end.
Dense prediction with convnets Several recent works
have applied convnets to dense prediction problems, including semantic segmentation by Ning et al. [27], Farabet et al.[8], and Pinheiro and Collobert [28]; boundary prediction
for electron microscopy by Ciresan et al. [2] and for natural
images by a hybrid neural net/nearest neighbor model by
Ganin and Lempitsky [11]; and image restoration and depth
estimation by Eigen et al. [5, 6]. Common elements of these
approaches include
small models restricting capacity and receptive ﬁelds;
patchwise training [27, 2, 8, 28, 11];
post-processing by superpixel projection, random ﬁeld

to appear in CVPR (2015)


patchwise training [27, 2, 8, 28, 11];
post-processing by superpixel projection, random ﬁeld
regularization, ﬁltering, or local classiﬁcation [8, 2,
11];
input shifting and output interlacing for dense output
[28, 11] as introduced by OverFeat [29];
multi-scale pyramid processing [8, 28, 11];
saturating tanh nonlinearities [8, 5, 28]; and
ensembles [2, 11],
whereas our method does without this machinery. However,
we do study patchwise training 3.4 and “shift-and-stitch”
dense output 3.2 from the perspective of FCNs. We also
discuss in-network upsampling 3.3, of which the fully connected prediction by Eigen et al. [6] is a special case.
Unlike these existing methods, we adapt and extend deep
classiﬁcation architectures, using image classiﬁcation as supervised pre-training, and ﬁne-tune fully convolutionally to
learn simply and efﬁciently from whole image inputs and
whole image ground thruths.

to appear in CVPR (2015)


learn simply and efﬁciently from whole image inputs and
whole image ground thruths.
Hariharan et al. [16] and Gupta et al. [14] likewise adapt
deep classiﬁcation nets to semantic segmentation, but do
so in hybrid proposal-classiﬁer models. These approaches
ﬁne-tune an R-CNN system [12] by sampling bounding
boxes and/or region proposals for detection, semantic segmentation, and instance segmentation. Neither method is
learned end-to-end.
They achieve state-of-the-art results on PASCAL VOC
segmentation and NYUDv2 segmentation respectively, so
we directly compare our standalone, end-to-end FCN to
their semantic segmentation results in Section 5.
3. Fully convolutional networks
Each layer of data in a convnet is a three-dimensional
array of size hwd, wherehandware spatial dimensions, anddis the feature or channel dimension. The ﬁrst
layer is the image, with pixel size hw, anddcolor channels. Locations in higher layers correspond to the locations
in the image they are path-connected to, which are called
their receptive ﬁelds .

to appear in CVPR (2015)


in the image they are path-connected to, which are called
their receptive ﬁelds .
Convnets are built on translation invariance. Their basic components (convolution, pooling, and activation functions) operate on local input regions, and depend only on
relative spatial coordinates. Writing xijfor the data vector
at location (i;j)in a particular layer, and yijfor the following layer, these functions compute outputs yijby
yij=fks(fxsi+i;sj +jg0i;jk)
wherekis called the kernel size, sis the stride or subsampling factor, and fksdetermines the layer type: a matrix
multiplication for convolution or average pooling, a spatial
max for max pooling, or an elementwise nonlinearity for an
activation function, and so on for other types of layers.
This functional form is maintained under composition,
with kernel size and stride obeying the transformation rule
fksgk0s0= (fg)k0+(k1)s0;ss0:
While a general deep net computes a general nonlinear
function, a net with only layers of this form computes a

to appear in CVPR (2015)


While a general deep net computes a general nonlinear
function, a net with only layers of this form computes a
nonlinear ﬁlter , which we call a deep ﬁlter orfully convolutional network . An FCN naturally operates on an input of
any size, and produces an output of corresponding (possibly
resampled) spatial dimensions.
A real-valued loss function composed with an FCN deﬁnes a task. If the loss function is a sum over the spatial
dimensions of the ﬁnal layer, `(x;) =P
ij`0(xij;), its
gradient will be a sum over the gradients of each of its spatial components. Thus stochastic gradient descent on `computed on whole images will be the same as stochastic gradient descent on `0, taking all of the ﬁnal layer receptive ﬁelds
as a minibatch.
When these receptive ﬁelds overlap signiﬁcantly, both
feedforward computation and backpropagation are much
more efﬁcient when computed layer-by-layer over an entire
image instead of independently patch-by-patch.
We next explain how to convert classiﬁcation nets into
fully convolutional nets that produce coarse output maps.

to appear in CVPR (2015)


image instead of independently patch-by-patch.
We next explain how to convert classiﬁcation nets into
fully convolutional nets that produce coarse output maps.
For pixelwise prediction, we need to connect these coarse
outputs back to the pixels. Section 3.2 describes a trick that
OverFeat [29] introduced for this purpose. We gain insight
into this trick by reinterpreting it as an equivalent network
modiﬁcation. As an efﬁcient, effective alternative, we introduce deconvolution layers for upsampling in Section 3.3.
In Section 3.4 we consider training by patchwise sampling,
and give evidence in Section 4.3 that our whole image training is faster and equally effective.
3.1. Adapting classiﬁers for dense prediction
Typical recognition nets, including LeNet [21], AlexNet
[19], and its deeper successors [31, 32], ostensibly take
ﬁxed-sized inputs and produce nonspatial outputs. The fully
connected layers of these nets have ﬁxed dimensions and
throw away spatial coordinates. However, these fully connected layers can also be viewed as convolutions with kernels that cover their entire input regions. Doing so casts
them into fully convolutional networks that take input of
any size and output classiﬁcation maps. This transformation

to appear in CVPR (2015)


them into fully convolutional networks that take input of
any size and output classiﬁcation maps. This transformation
`tabby cat" `
96256384384256409640961000
96384256409640961000
256384tabby cat heatmapconvolutionalizationFigure 2. Transforming fully connected layers into convolution
layers enables a classiﬁcation net to output a heatmap. Adding
layers and a spatial loss (as in Figure 1) produces an efﬁcient machine for end-to-end dense learning.
is illustrated in Figure 2. (By contrast, nonconvolutional
nets, such as the one by Le et al. [20], lack this capability.)
Furthermore, while the resulting maps are equivalent to
the evaluation of the original net on particular input patches,
the computation is highly amortized over the overlapping
regions of those patches. For example, while AlexNet takes
1:2ms (on a typical GPU) to produce the classiﬁcation
scores of a 227227image, the fully convolutional version takes 22ms to produce a 1010grid of outputs from
a500500image, which is more than 5times faster than
the na ¨ıve approach1.

to appear in CVPR (2015)


a500500image, which is more than 5times faster than
the na ¨ıve approach1.
The spatial output maps of these convolutionalized models make them a natural choice for dense problems like semantic segmentation. With ground truth available at every output cell, both the forward and backward passes are
straightforward, and both take advantage of the inherent
computational efﬁciency (and aggressive optimization) of
convolution.
The corresponding backward times for the AlexNet example are 2:4ms for a single image and 37ms for a fully
convolutional 1010output map, resulting in a speedup
similar to that of the forward pass. This dense backpropagation is illustrated in Figure 1.
While our reinterpretation of classiﬁcation nets as fully
convolutional yields output maps for inputs of any size, the
output dimensions are typically reduced by subsampling.
The classiﬁcation nets subsample to keep ﬁlters small and
computational requirements reasonable. This coarsens the
output of a fully convolutional version of these nets, reducing it from the size of the input by a factor equal to the pixel
stride of the receptive ﬁelds of the output units.
1Assuming efﬁcient batching of single image inputs. The classiﬁcation

to appear in CVPR (2015)


stride of the receptive ﬁelds of the output units.
1Assuming efﬁcient batching of single image inputs. The classiﬁcation
scores for a single image by itself take 5.4 ms to produce, which is nearly
25times slower than the fully convolutional version.
3.2. Shift-and-stitch is ﬁlter rarefaction
Input shifting and output interlacing is a trick that yields
dense predictions from coarse outputs without interpolation, introduced by OverFeat [29]. If the outputs are downsampled by a factor of f, the input is shifted (by left and top
padding)xpixels to the right and ypixels down, once for
every value of (x;y)2f0;:::;f1gf 0;:::;f1g.
Thesef2inputs are each run through the convnet, and the
outputs are interlaced so that the predictions correspond to
the pixels at the centers of their receptive ﬁelds.
Changing only the ﬁlters and layer strides of a convnet
can produce the same output as this shift-and-stitch trick.
Consider a layer (convolution or pooling) with input stride

to appear in CVPR (2015)


can produce the same output as this shift-and-stitch trick.
Consider a layer (convolution or pooling) with input stride
s, and a following convolution layer with ﬁlter weights fij
(eliding the feature dimensions, irrelevant here). Setting the
lower layer’s input stride to 1upsamples its output by a factor ofs, just like shift-and-stitch. However, convolving the
original ﬁlter with the upsampled output does not produce
the same result as the trick, because the original ﬁlter only
sees a reduced portion of its (now upsampled) input. To
reproduce the trick, rarefy the ﬁlter by enlarging it as
f0
ij=fi=s;j=s ifsdivides both iandj;
0 otherwise,
(withiandjzero-based). Reproducing the full net output
of the trick involves repeating this ﬁlter enlargement layerby-layer until all subsampling is removed.
Simply decreasing subsampling within a net is a tradeoff:
the ﬁlters see ﬁner information, but have smaller receptive
ﬁelds and take longer to compute. We have seen that the
shift-and-stitch trick is another kind of tradeoff: the output

to appear in CVPR (2015)


ﬁelds and take longer to compute. We have seen that the
shift-and-stitch trick is another kind of tradeoff: the output
is made denser without decreasing the receptive ﬁeld sizes
of the ﬁlters, but the ﬁlters are prohibited from accessing
information at a ﬁner scale than their original design.
Although we have done preliminary experiments with
shift-and-stitch, we do not use it in our model. We ﬁnd
learning through upsampling, as described in the next section, to be more effective and efﬁcient, especially when
combined with the skip layer fusion described later on.
3.3. Upsampling is backwards strided convolution
Another way to connect coarse outputs to dense pixels
is interpolation. For instance, simple bilinear interpolation
computes each output yijfrom the nearest four inputs by a
linear map that depends only on the relative positions of the
input and output cells.
In a sense, upsampling with factor fis convolution with
afractional input stride of 1=f. So long as fis integral, a
natural way to upsample is therefore backwards convolution
(sometimes called deconvolution ) with an output stride of
f. Such an operation is trivial to implement, since it simply

to appear in CVPR (2015)


natural way to upsample is therefore backwards convolution
(sometimes called deconvolution ) with an output stride of
f. Such an operation is trivial to implement, since it simply
reverses the forward and backward passes of convolution.Thus upsampling is performed in-network for end-to-end
learning by backpropagation from the pixelwise loss.
Note that the deconvolution ﬁlter in such a layer need not
be ﬁxed (e.g., to bilinear upsampling), but can be learned.
A stack of deconvolution layers and activation functions can
even learn a nonlinear upsampling.
In our experiments, we ﬁnd that in-network upsampling
is fast and effective for learning dense prediction. Our best
segmentation architecture uses these layers to learn to upsample for reﬁned prediction in Section 4.2.
3.4. Patchwise training is loss sampling
In stochastic optimization, gradient computation is
driven by the training distribution. Both patchwise training and fully-convolutional training can be made to produce any distribution, although their relative computational
efﬁciency depends on overlap and minibatch size. Whole
image fully convolutional training is identical to patchwise
training where each batch consists of all the receptive ﬁelds

to appear in CVPR (2015)


image fully convolutional training is identical to patchwise
training where each batch consists of all the receptive ﬁelds
of the units below the loss for an image (or collection of
images). While this is more efﬁcient than uniform sampling
of patches, it reduces the number of possible batches. However, random selection of patches within an image may be
recovered simply. Restricting the loss to a randomly sampled subset of its spatial terms (or, equivalently applying a
DropConnect mask [36] between the output and the loss)
excludes patches from the gradient computation.
If the kept patches still have signiﬁcant overlap, fully
convolutional computation will still speed up training. If
gradients are accumulated over multiple backward passes,
batches can include patches from several images.2
Sampling in patchwise training can correct class imbalance [27, 8, 2] and mitigate the spatial correlation of dense
patches [28, 16]. In fully convolutional training, class balance can also be achieved by weighting the loss, and loss
sampling can be used to address spatial correlation.
We explore training with sampling in Section 4.3, and do
not ﬁnd that it yields faster or better convergence for dense
prediction. Whole image training is effective and efﬁcient.
4. Segmentation Architecture

to appear in CVPR (2015)


not ﬁnd that it yields faster or better convergence for dense
prediction. Whole image training is effective and efﬁcient.
4. Segmentation Architecture
We cast ILSVRC classiﬁers into FCNs and augment
them for dense prediction with in-network upsampling and
a pixelwise loss. We train for segmentation by ﬁne-tuning.
Next, we build a novel skip architecture that combines
coarse, semantic and local, appearance information to reﬁne prediction.
For this investigation, we train and validate on the PASCAL VOC 2011 segmentation challenge [7]. We train with
2Note that not every possible patch is included this way, since the receptive ﬁelds of the ﬁnal layer units lie on a ﬁxed, strided grid. However,
by shifting the image left and down by a random value up to the stride,
random selection from all possible patches may be recovered.
a per-pixel multinomial logistic loss and validate with the
standard metric of mean pixel intersection over union, with
the mean taken over all classes, including background. The
training ignores pixels that are masked out (as ambiguous
or difﬁcult) in the ground truth.
4.1. From classiﬁer to dense FCN

to appear in CVPR (2015)


training ignores pixels that are masked out (as ambiguous
or difﬁcult) in the ground truth.
4.1. From classiﬁer to dense FCN
We begin by convolutionalizing proven classiﬁcation architectures as in Section 3. We consider the AlexNet3architecture [19] that won ILSVRC12, as well as the VGG
nets [31] and the GoogLeNet4[32] which did exceptionally well in ILSVRC14. We pick the VGG 16-layer net5,
which we found to be equivalent to the 19-layer net on this
task. For GoogLeNet, we use only the ﬁnal loss layer, and
improve performance by discarding the ﬁnal average pooling layer. We decapitate each net by discarding the ﬁnal
classiﬁer layer, and convert all fully connected layers to
convolutions. We append a 11convolution with channel dimension 21to predict scores for each of the PASCAL classes (including background) at each of the coarse
output locations, followed by a deconvolution layer to bilinearly upsample the coarse outputs to pixel-dense outputs
as described in Section 3.3. Table 1 compares the preliminary validation results along with the basic characteristics

to appear in CVPR (2015)


as described in Section 3.3. Table 1 compares the preliminary validation results along with the basic characteristics
of each net. We report the best results achieved after convergence at a ﬁxed learning rate (at least 175 epochs).
Fine-tuning from classiﬁcation to segmentation gave reasonable predictions for each net. Even the worst model
achieved75% of state-of-the-art performance. The
segmentation-equippped VGG net (FCN-VGG16) already
appears to be state-of-the-art at 56.0 mean IU on val, compared to 52.6 on test [16]. Training on extra data raises
performance to 59.4 mean IU on a subset of val7. Training
details are given in Section 4.3.
Despite similar classiﬁcation accuracy, our implementation of GoogLeNet did not match this segmentation result.
4.2. Combining what and where
We deﬁne a new fully convolutional net (FCN) for segmentation that combines layers of the feature hierarchy and
reﬁnes the spatial precision of the output. See Figure 3.
While fully convolutionalized classiﬁers can be ﬁnetuned to segmentation as shown in 4.1, and even score

to appear in CVPR (2015)


While fully convolutionalized classiﬁers can be ﬁnetuned to segmentation as shown in 4.1, and even score
highly on the standard metric, their output is dissatisfyingly
coarse (see Figure 4). The 32 pixel stride at the ﬁnal prediction layer limits the scale of detail in the upsampled output.
We address this by adding links that combine the ﬁnal
prediction layer with lower layers with ﬁner strides. This
3Using the publicly available CaffeNet reference model.
4Since there is no publicly available version of GoogLeNet, we use
our own reimplementation. Our version is trained with less extensive data
augmentation, and gets 68.5% top-1 and 88.4% top-5 ILSVRC accuracy.
5Using the publicly available version from the Caffe model zoo.Table 1. We adapt and extend three classiﬁcation convnets to segmentation. We compare performance by mean intersection over
union on the validation set of PASCAL VOC 2011 and by inference time (averaged over 20 trials for a 500500 input on an
NVIDIA Tesla K40c). We detail the architecture of the adapted
nets as regards dense prediction: number of parameter layers, receptive ﬁeld size of output units, and the coarsest stride within the

to appear in CVPR (2015)


nets as regards dense prediction: number of parameter layers, receptive ﬁeld size of output units, and the coarsest stride within the
net. (These numbers give the best performance obtained at a ﬁxed
learning rate, not best performance possible.)
FCNAlexNetFCNVGG16FCNGoogLeNet4
mean IU 39.8 56.0 42.5
forward time 50 ms 210 ms 59 ms
conv. layers 8 16 22
parameters 57M 134M 6M
rf size 355 404 907
max stride 32 32 32
turns a line topology into a DAG, with edges that skip ahead
from lower layers to higher ones (Figure 3). As they see
fewer pixels, the ﬁner scale predictions should need fewer
layers, so it makes sense to make them from shallower net
outputs. Combining ﬁne layers and coarse layers lets the
model make local predictions that respect global structure.
By analogy to the multiscale local jet of Florack et al. [10],
we call our nonlinear local feature hierarchy the deep jet .
We ﬁrst divide the output stride in half by predicting
from a 16 pixel stride layer. We add a 11convolution
layer on top of pool4 to produce additional class predictions. We fuse this output with the predictions computed

to appear in CVPR (2015)


from a 16 pixel stride layer. We add a 11convolution
layer on top of pool4 to produce additional class predictions. We fuse this output with the predictions computed
on top of conv7 (convolutionalized fc7) at stride 32 by
adding a 2upsampling layer and summing6both predictions. (See Figure 3). We initialize the 2upsampling to
bilinear interpolation, but allow the parameters to be learned
as described in Section 3.3. Finally, the stride 16 predictions
are upsampled back to the image. We call this net FCN-16s.
FCN-16s is learned end-to-end, initialized with the parameters of the last, coarser net, which we now call FCN-32s.
The new parameters acting on pool4 are zero-initialized so
that the net starts with unmodiﬁed predictions. The learning
rate is decreased by a factor of 100.
Learning this skip net improves performance on the validation set by 3.0 mean IU to 62.4. Figure 4 shows improvement in the ﬁne structure of the output. We compared
this fusion with learning only from the pool4 layer (which
resulted in poor performance), and simply decreasing the
learning rate without adding the extra link (which results

to appear in CVPR (2015)


this fusion with learning only from the pool4 layer (which
resulted in poor performance), and simply decreasing the
learning rate without adding the extra link (which results
in an insigniﬁcant performance improvement, without improving the quality of the output).
We continue in this fashion by fusing predictions from
pool3 with a 2upsampling of predictions fused from
pool4 andconv7 , building the net FCN-8s. We obtain
6Max fusion made learning difﬁcult due to gradient switching.
FCN-32s FCN-16s FCN-8s Ground truth
Figure 4. Reﬁning fully convolutional nets by fusing information
from layers with different strides improves segmentation detail.
The ﬁrst three images show the output from our 32, 16, and 8
pixel stride nets (see Figure 3).
Table 2. Comparison of skip FCNs on a subset of PASCAL
VOC2011 validation7. Learning is end-to-end, except for FCN32s-ﬁxed, where only the last layer is ﬁne-tuned. Note that FCN32s is FCN-VGG16, renamed to highlight stride.
pixel
acc.mean
acc.mean
IUf.w.
IU

to appear in CVPR (2015)


pixel
acc.mean
acc.mean
IUf.w.
IU
FCN-32s-ﬁxed 83.0 59.7 45.4 72.0
FCN-32s 89.1 73.3 59.4 81.4
FCN-16s 90.0 75.7 62.4 83.0
FCN-8s 90.3 75.9 62.7 83.2
a minor additional improvement to 62.7 mean IU, and ﬁnd
a slight improvement in the smoothness and detail of our
output. At this point our fusion improvements have met diminishing returns, both with respect to the IU metric which
emphasizes large-scale correctness, and also in terms of the
improvement visible e.g. in Figure 4, so we do not continue
fusing even lower layers.
Reﬁnement by other means Decreasing the stride of
pooling layers is the most straightforward way to obtain
ﬁner predictions. However, doing so is problematic for our
VGG16-based net. Setting the pool5 layer to have stride 1
requires our convolutionalized fc6 to have a kernel size of1414in order to maintain its receptive ﬁeld size. In addition to their computational cost, we had difﬁculty learning

to appear in CVPR (2015)


such large ﬁlters. We made an attempt to re-architect the
layers above pool5 with smaller ﬁlters, but were not successful in achieving comparable performance; one possible
explanation is that the initialization from ImageNet-trained
weights in the upper layers is important.
Another way to obtain ﬁner predictions is to use the shiftand-stitch trick described in Section 3.2. In limited experiments, we found the cost to improvement ratio from this
method to be worse than layer fusion.
4.3. Experimental framework
Optimization We train by SGD with momentum. We
use a minibatch size of 20 images and ﬁxed learning rates of
103,104, and 55for FCN-AlexNet, FCN-VGG16, and
FCN-GoogLeNet, respectively, chosen by line search. We
use momentum 0:9, weight decay of 54or24, and doubled the learning rate for biases, although we found training
to be insensitive to these parameters (but sensitive to the
learning rate). We zero-initialize the class scoring convolution layer, ﬁnding random initialization to yield neither
better performance nor faster convergence. Dropout was included where used in the original classiﬁer nets.

to appear in CVPR (2015)


better performance nor faster convergence. Dropout was included where used in the original classiﬁer nets.
Fine-tuning We ﬁne-tune all layers by backpropagation through the whole net. Fine-tuning the
output classiﬁer alone yields only 70% of the full ﬁnetuning performance as compared in Table 2. Training from
scratch is not feasible considering the time required to
learn the base classiﬁcation nets. (Note that the VGG net is
trained in stages, while we initialize from the full 16-layer
version.) Fine-tuning takes three days on a single GPU for
the coarse FCN-32s version, and about one day each to
upgrade to the FCN-16s and FCN-8s versions.
Patch Sampling As explained in Section 3.4, our full
image training effectively batches each image into a reguimage pool4 pool5 pool1 pool2 pool332x upsampled
prediction (FCN-32s)2x upsampled
prediction16x upsampled
prediction (FCN-16s)8x upsampled
prediction (FCN-8s)
pool4
prediction2x upsampled
prediction
pool3
predictionP P

to appear in CVPR (2015)


prediction (FCN-8s)
pool4
prediction2x upsampled
prediction
pool3
predictionP P
Figure 3. Our DAG nets learn to combine coarse, high layer information with ﬁne, low layer information. Layers are shown as grids that
reveal relative spatial coarseness. Only pooling and prediction layers are shown; intermediate convolution layers (including our converted
fully connected layers) are omitted. Solid line (FCN-32s): Our single-stream net, described in Section 4.1, upsamples stride 32 predictions
back to pixels in a single step. Dashed line (FCN-16s): Combining predictions from both the ﬁnal layer and the pool4 layer, at stride
16, lets our net predict ﬁner details, while retaining high-level semantic information. Dotted line (FCN-8s): Additional predictions from
pool3 , at stride 8, provide further precision.
500 1000 1500
iteration number0.40.60.81.01.2lossfull images
50% sampling
25% sampling
10000 20000 30000
relative time (num. images processed)0.40.60.81.01.2lossFigure 5. Training on whole images is just as effective as sampling
patches, but results in faster (wall time) convergence by making

to appear in CVPR (2015)


patches, but results in faster (wall time) convergence by making
more efﬁcient use of data. Left shows the effect of sampling on
convergence rate for a ﬁxed expected batch size, while right plots
the same by relative wall time.
lar grid of large, overlapping patches. By contrast, prior
work randomly samples patches over a full dataset [27, 2,
8, 28, 11], potentially resulting in higher variance batches
that may accelerate convergence [22]. We study this tradeoff by spatially sampling the loss in the manner described
earlier, making an independent choice to ignore each ﬁnal
layer cell with some probability 1p. To avoid changing the
effective batch size, we simultaneously increase the number
of images per batch by a factor 1=p. Note that due to the efﬁciency of convolution, this form of rejection sampling is
still faster than patchwise training for large enough values
ofp(e.g., at least for p > 0:2according to the numbers
in Section 3.1). Figure 5 shows the effect of this form of
sampling on convergence. We ﬁnd that sampling does not
have a signiﬁcant effect on convergence rate compared to
whole image training, but takes signiﬁcantly more time due

to appear in CVPR (2015)


have a signiﬁcant effect on convergence rate compared to
whole image training, but takes signiﬁcantly more time due
to the larger number of images that need to be considered
per batch. We therefore choose unsampled, whole image
training in our other experiments.
Class Balancing Fully convolutional training can balance classes by weighting or sampling the loss. Although
our labels are mildly unbalanced (about 3=4are background), we ﬁnd class balancing unnecessary.
Dense Prediction The scores are upsampled to the input dimensions by deconvolution layers within the net. Final layer deconvolutional ﬁlters are ﬁxed to bilinear interpolation, while intermediate upsampling layers are initialized to bilinear upsampling, and then learned. Shift-andstitch (Section 3.2), or the ﬁlter rarefaction equivalent, are
not used.
Augmentation We tried augmenting the training data
by randomly mirroring and “jittering” the images by translating them up to 32 pixels (the coarsest scale of prediction)
in each direction. This yielded no noticeable improvement.
More Training Data The PASCAL VOC 2011 segmentation challenge training set, which we used for Table 1,

to appear in CVPR (2015)


in each direction. This yielded no noticeable improvement.
More Training Data The PASCAL VOC 2011 segmentation challenge training set, which we used for Table 1,
labels 1112 images. Hariharan et al . [15] have collectedlabels for a much larger set of 8498 PASCAL training images, which was used to train the previous state-of-the-art
system, SDS [16]. This training data improves the FCNVGG16 validation score7by 3.4 points to 59.4 mean IU.
Implementation All models are trained and tested with
Caffe [18] on a single NVIDIA Tesla K40c. The models
and code will be released open-source on publication.
5. Results
We test our FCN on semantic segmentation and scene
parsing, exploring PASCAL VOC, NYUDv2, and SIFT
Flow. Although these tasks have historically distinguished
between objects and regions, we treat both uniformly as
pixel prediction. We evaluate our FCN skip architecture8
on each of these datasets, and then extend it to multi-modal
input for NYUDv2 and multi-task prediction for the semantic and geometric labels of SIFT Flow.
Metrics We report four metrics from common semantic
segmentation and scene parsing evaluations that are variations on pixel accuracy and region intersection over union

to appear in CVPR (2015)


Metrics We report four metrics from common semantic
segmentation and scene parsing evaluations that are variations on pixel accuracy and region intersection over union
(IU). Letnijbe the number of pixels of class ipredicted to
belong to class j, where there are ncldifferent classes, and
letti=P
jnijbe the total number of pixels of class i. We
compute:
pixel accuracy:P
inii=P
iti
mean accuraccy: (1=ncl)P
inii=ti
mean IU: (1=ncl)P
inii=
ti+P
jnjinii
frequency weighted IU:
(P
ktk)1P
itinii=
ti+P
jnjinii
PASCAL VOC Table 3 gives the performance of our
FCN-8s on the test sets of PASCAL VOC 2011 and 2012,
and compares it to the previous state-of-the-art, SDS [16],
and the well-known R-CNN [12]. We achieve the best results on mean IU9by a relative margin of 20%. Inference
time is reduced 114(convnet only, ignoring proposals and
reﬁnement) or 286(overall).

to appear in CVPR (2015)


time is reduced 114(convnet only, ignoring proposals and
reﬁnement) or 286(overall).
Table 3. Our fully convolutional net gives a 20% relative improvement over the state-of-the-art on the PASCAL VOC 2011 and 2012
test sets, and reduces inference time.
mean IU mean IU inference
VOC2011 test VOC2012 test time
R-CNN [12] 47.9 - SDS [16] 52.6 51.6 50 s
FCN-8s 62.7 62.2 175 ms
NYUDv2 [30] is an RGB-D dataset collected using the
7There are training images from [15] included in the PASCAL VOC
2011 val set, so we validate on the non-intersecting set of 736 images. An
earlier version of this paper mistakenly evaluated on the entire val set.
8Our models and code are publicly available at
https://github.com/BVLC/caffe/wiki/Model-Zoo#fcn .
9This is the only metric provided by the test server.
Table 4. Results on NYUDv2. RGBD is early-fusion of the
RGB and depth channels at the input. HHA is the depth embedding of [14] as horizontal disparity, height above ground, and

to appear in CVPR (2015)


RGB and depth channels at the input. HHA is the depth embedding of [14] as horizontal disparity, height above ground, and
the angle of the local surface normal with the inferred gravity
direction. RGB-HHA is the jointly trained late fusion model
that sums RGB and HHA predictions.
pixel
acc.mean
acc.mean
IUf.w.
IU
Gupta et al. [14] 60.3 - 28.6 47.0
FCN-32s RGB 60.0 42.2 29.2 43.9
FCN-32s RGBD 61.5 42.4 30.5 45.5
FCN-32s HHA 57.1 35.2 24.2 40.4
FCN-32s RGB-HHA 64.3 44.9 32.8 48.0
FCN-16s RGB-HHA 65.4 46.1 34.0 49.5
Microsoft Kinect. It has 1449 RGB-D images, with pixelwise labels that have been coalesced into a 40 class semantic segmentation task by Gupta et al. [13]. We report results
on the standard split of 795 training images and 654 testing
images. (Note: all model selection is performed on PASCAL 2011 val.) Table 4 gives the performance of our model

to appear in CVPR (2015)


on the standard split of 795 training images and 654 testing
images. (Note: all model selection is performed on PASCAL 2011 val.) Table 4 gives the performance of our model
in several variations. First we train our unmodiﬁed coarse
model (FCN-32s) on RGB images. To add depth information, we train on a model upgraded to take four-channel
RGB-D input (early fusion). This provides little beneﬁt,
perhaps due to the difﬁcultly of propagating meaningful
gradients all the way through the model. Following the success of Gupta et al. [14], we try the three-dimensional HHA
encoding of depth, training nets on just this information, as
well as a “late fusion” of RGB and HHA where the predictions from both nets are summed at the ﬁnal layer, and the
resulting two-stream net is learned end-to-end. Finally we
upgrade this late fusion net to a 16-stride version.
SIFT Flow is a dataset of 2,688 images with pixel labels
for 33 semantic categories (“bridge”, “mountain”, “sun”),

to appear in CVPR (2015)


SIFT Flow is a dataset of 2,688 images with pixel labels
for 33 semantic categories (“bridge”, “mountain”, “sun”),
as well as three geometric categories (“horizontal”, “vertical”, and “sky”). An FCN can naturally learn a joint representation that simultaneously predicts both types of labels.
We learn a two-headed version of FCN-16s with semantic and geometric prediction layers and losses. The learned
model performs as well on both tasks as two independently
trained models, while learning and inference are essentially
as fast as each independent model by itself. The results in
Table 5, computed on the standard split into 2,488 training
and 200 test images,10show state-of-the-art performance on
both tasks.
10Three of the SIFT Flow categories are not present in the test set. We
made predictions across all 33 categories, but only included categories actually present in the test set in our evaluation. (An earlier version of this paper reported a lower mean IU, which included all categories either present
or predicted in the evaluation.)Table 5. Results on SIFT Flow10with class segmentation
(center) and geometric segmentation (right). Tighe [33] is
a non-parametric transfer method. Tighe 1 is an exemplar

to appear in CVPR (2015)


(center) and geometric segmentation (right). Tighe [33] is
a non-parametric transfer method. Tighe 1 is an exemplar
SVM while 2 is SVM + MRF. Farabet is a multi-scale convnet trained on class-balanced samples (1) or natural frequency
samples (2). Pinheiro is a multi-scale, recurrent convnet, denoted RCNN 3(3). The metric for geometry is pixel accuracy.
pixel
acc.mean
acc.mean
IUf.w.
IUgeom.
acc.
Liuet al. [23] 76.7 - - - Tighe et al. [33] - - - - 90.8
Tighe et al. [34] 1 75.6 41.1 - - Tighe et al. [34] 2 78.6 39.2 - - Farabet et al. [8] 1 72.3 50.8 - - Farabet et al. [8] 2 78.5 29.6 - - Pinheiro et al. [28] 77.7 29.8 - - FCN-16s 85.2 51.7 39.5 76.1 94.3
FCN-8s SDS [16] Ground Truth Image

to appear in CVPR (2015)


FCN-8s SDS [16] Ground Truth Image
Figure 6. Fully convolutional segmentation nets produce stateof-the-art performance on PASCAL. The left column shows the
output of our highest performing net, FCN-8s. The second shows
the segmentations produced by the previous state-of-the-art system
by Hariharan et al. [16]. Notice the ﬁne structures recovered (ﬁrst
row), ability to separate closely interacting objects (second row),
and robustness to occluders (third row). The fourth row shows a
failure case: the net sees lifejackets in a boat as people.
6. Conclusion
Fully convolutional networks are a rich class of models, of which modern classiﬁcation convnets are a special case. Recognizing this, extending these classiﬁcation
nets to segmentation, and improving the architecture with
multi-resolution layer combinations dramatically improves
the state-of-the-art, while simultaneously simplifying and
speeding up learning and inference.
Acknowledgements This work was supported in part
by DARPA’s MSEE and SMISC programs, NSF awards IIS1427425, IIS-1212798, IIS-1116411, and the NSF GRFP,

to appear in CVPR (2015)


Toyota, and the Berkeley Vision and Learning Center. We
gratefully acknowledge NVIDIA for GPU donation. We
thank Bharath Hariharan and Saurabh Gupta for their advice and dataset tools. We thank Sergio Guadarrama for
reproducing GoogLeNet in Caffe. We thank Jitendra Malik
for his helpful comments. Thanks to Wei Liu for pointing
out an issue wth our SIFT Flow mean IU computation and
an error in our frequency weighted mean IU formula.
A. Upper Bounds on IU
In this paper, we have achieved good performance on
the mean IU segmentation metric even with coarse semantic
prediction. To better understand this metric and the limits
of this approach with respect to it, we compute approximate
upper bounds on performance with prediction at various
scales. We do this by downsampling ground truth images
and then upsampling them again to simulate the best results
obtainable with a particular downsampling factor. The following table gives the mean IU on a subset of PASCAL
2011 val for various downsampling factors.
factor mean IU
128 50.9
64 73.3
32 86.1
16 92.8
896.4
498.5
Pixel-perfect prediction is clearly not necessary to

to appear in CVPR (2015)


128 50.9
64 73.3
32 86.1
16 92.8
896.4
498.5
Pixel-perfect prediction is clearly not necessary to
achieve mean IU well above state-of-the-art, and, conversely, mean IU is a not a good measure of ﬁne-scale accuracy.
B. More Results
We further evaluate our FCN for semantic segmentation.
PASCAL-Context [26] provides whole scene annotations of PASCAL VOC 2010. While there are over 400 distinct classes, we follow the 59 class task deﬁned by [26] that
picks the most frequent classes. We train and evaluate on
the training and val sets respectively. In Table 6, we compare to the joint object + stuff variation of Convolutional
Feature Masking [3] which is the previous state-of-the-art
on this task. FCN-8s scores 35.1 mean IU for an 11% relative improvement.
Changelog
The arXiv version of this paper is kept up-to-date with
corrections and additional relevant material. The following
gives a brief history of changes.Table 6. Results on PASCAL-Context. CFM is the best result of
[3] by convolutional feature masking and segment pursuit with the

to appear in CVPR (2015)


[3] by convolutional feature masking and segment pursuit with the
VGG net. O2Pis the second order pooling method [1] as reported
in the errata of [26]. The 59 class task includes the 59 most frequent classes while the 33 class task consists of an easier subset
identiﬁed by [26].
59 classpixel
acc.mean
acc.mean
IUf.w.
IU
O2P - - 18.1 CFM - - 31.5 FCN-32s 63.8 42.7 31.8 48.3
FCN-16s 65.7 46.2 34.8 50.7
FCN-8s 65.9 46.5 35.1 51.0
33 class
O2P - - 29.2 CFM - - 46.1 FCN-32s 69.8 65.1 50.4 54.9
FCN-16s 71.8 68.0 53.4 57.5
FCN-8s 71.8 67.6 53.5 57.7
v2Add Appendix A giving upper bounds on mean IU and
Appendix B with PASCAL-Context results. Correct PASCAL validation numbers (previously, some val images were
included in train), SIFT Flow mean IU (which used an inappropriately strict metric), and an error in the frequency

to appear in CVPR (2015)


included in train), SIFT Flow mean IU (which used an inappropriately strict metric), and an error in the frequency
weighted mean IU formula. Add link to models and update
timing numbers to reﬂect improved implementation (which
is publicly available).
References
[1] J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu. Semantic segmentation with second-order pooling. In ECCV ,
2012. 9
[2] D. C. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber. Deep neural networks segment neuronal membranes
in electron microscopy images. In NIPS , pages 2852–2860,
2012. 1, 2, 4, 7
[3] J. Dai, K. He, and J. Sun. Convolutional feature masking for joint object and stuff segmentation. arXiv preprint
arXiv:1412.1283 , 2014. 9
[4] J. Donahue, Y . Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. DeCAF: A deep convolutional activation feature for generic visual recognition. In ICML , 2014.
1, 2

to appear in CVPR (2015)


E. Tzeng, and T. Darrell. DeCAF: A deep convolutional activation feature for generic visual recognition. In ICML , 2014.
1, 2
[5] D. Eigen, D. Krishnan, and R. Fergus. Restoring an image
taken through a window covered with dirt or rain. In Computer Vision (ICCV), 2013 IEEE International Conference
on, pages 633–640. IEEE, 2013. 2
[6] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction
from a single image using a multi-scale deep network. arXiv
preprint arXiv:1406.2283 , 2014. 2
[7] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The PASCAL Visual Object Classes
Challenge 2011 (VOC2011) Results. http://www.pascalnetwork.org/challenges/VOC/voc2011/workshop/index.html.
4
[8] C. Farabet, C. Couprie, L. Najman, and Y . LeCun. Learning
hierarchical features for scene labeling. Pattern Analysis and
Machine Intelligence, IEEE Transactions on , 2013. 1, 2, 4,
7, 8

to appear in CVPR (2015)


hierarchical features for scene labeling. Pattern Analysis and
Machine Intelligence, IEEE Transactions on , 2013. 1, 2, 4,
7, 8
[9] P. Fischer, A. Dosovitskiy, and T. Brox. Descriptor matching
with convolutional neural networks: a comparison to SIFT.
CoRR , abs/1405.5769, 2014. 1
[10] L. Florack, B. T. H. Romeny, M. Viergever, and J. Koenderink. The gaussian scale-space paradigm and the multiscale local jet. International Journal of Computer Vision ,
18(1):61–75, 1996. 5
[11] Y . Ganin and V . Lempitsky. N4-ﬁelds: Neural network nearest neighbor ﬁelds for image transforms. In ACCV , 2014. 1,
2, 7
[12] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In Computer Vision and Pattern Recognition ,
2014. 1, 2, 7
[13] S. Gupta, P. Arbelaez, and J. Malik. Perceptual organization
and recognition of indoor scenes from RGB-D images. In

to appear in CVPR (2015)


[13] S. Gupta, P. Arbelaez, and J. Malik. Perceptual organization
and recognition of indoor scenes from RGB-D images. In
CVPR , 2013. 8
[14] S. Gupta, R. Girshick, P. Arbelaez, and J. Malik. Learning
rich features from RGB-D images for object detection and
segmentation. In ECCV . Springer, 2014. 1, 2, 8
[15] B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik.
Semantic contours from inverse detectors. In International
Conference on Computer Vision (ICCV) , 2011. 7
[16] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision (ECCV) , 2014. 1, 2, 4, 5, 7, 8
[17] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling
in deep convolutional networks for visual recognition. In
ECCV , 2014. 1, 2

to appear in CVPR (2015)


in deep convolutional networks for visual recognition. In
ECCV , 2014. 1, 2
[18] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint
arXiv:1408.5093 , 2014. 7
[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
NIPS , 2012. 1, 2, 3, 5
[20] Q. V . Le, R. Monga, M. Devin, K. Chen, G. S. Corrado,
J. Dean, and A. Y . Ng. Building high-level features using
large scale unsupervised learning. In ICML , 2012. 3
[21] Y . LeCun, B. Boser, J. Denker, D. Henderson, R. E. Howard,
W. Hubbard, and L. D. Jackel. Backpropagation applied to
hand-written zip code recognition. In Neural Computation ,
1989. 2, 3

to appear in CVPR (2015)


W. Hubbard, and L. D. Jackel. Backpropagation applied to
hand-written zip code recognition. In Neural Computation ,
1989. 2, 3
[22] Y . A. LeCun, L. Bottou, G. B. Orr, and K.-R. M ¨uller. Efﬁcient backprop. In Neural networks: Tricks of the trade ,
pages 9–48. Springer, 1998. 7
[23] C. Liu, J. Yuen, and A. Torralba. Sift ﬂow: Dense correspondence across scenes and its applications. Pattern Analysis
and Machine Intelligence, IEEE Transactions on , 33(5):978–
994, 2011. 8[24] J. Long, N. Zhang, and T. Darrell. Do convnets learn correspondence? In NIPS , 2014. 1
[25] O. Matan, C. J. Burges, Y . LeCun, and J. S. Denker. Multidigit recognition using a space displacement neural network.
InNIPS , pages 488–495. Citeseer, 1991. 2

to appear in CVPR (2015)


InNIPS , pages 488–495. Citeseer, 1991. 2
[26] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object
detection and semantic segmentation in the wild. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE
Conference on , pages 891–898. IEEE, 2014. 9
[27] F. Ning, D. Delhomme, Y . LeCun, F. Piano, L. Bottou, and
P. E. Barbano. Toward automatic phenotyping of developing
embryos from videos. Image Processing, IEEE Transactions
on, 14(9):1360–1371, 2005. 1, 2, 4, 7
[28] P. H. Pinheiro and R. Collobert. Recurrent convolutional
neural networks for scene labeling. In ICML , 2014. 1, 2,
4, 7, 8
[29] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization

to appear in CVPR (2015)


and Y . LeCun. Overfeat: Integrated recognition, localization
and detection using convolutional networks. In ICLR , 2014.
1, 2, 3, 4
[30] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor
segmentation and support inference from rgbd images. In
ECCV , 2012. 7
[31] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR ,
abs/1409.1556, 2014. 1, 2, 3, 5
[32] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. CoRR , abs/1409.4842,
2014. 1, 2, 3, 5
[33] J. Tighe and S. Lazebnik. Superparsing: scalable nonparametric image parsing with superpixels. In ECCV , pages 352–
365. Springer, 2010. 8
[34] J. Tighe and S. Lazebnik. Finding things: Image parsing with

to appear in CVPR (2015)


365. Springer, 2010. 8
[34] J. Tighe and S. Lazebnik. Finding things: Image parsing with
regions and per-exemplar detectors. In CVPR , 2013. 8
[35] J. Tompson, A. Jain, Y . LeCun, and C. Bregler. Joint training
of a convolutional network and a graphical model for human
pose estimation. CoRR , abs/1406.2984, 2014. 2
[36] L. Wan, M. Zeiler, S. Zhang, Y . L. Cun, and R. Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine
Learning (ICML-13) , pages 1058–1066, 2013. 4
[37] R. Wolf and J. C. Platt. Postal address block location using
a convolutional locator network. Advances in Neural Information Processing Systems , pages 745–745, 1994. 2
[38] M. D. Zeiler and R. Fergus. Visualizing and understanding
convolutional networks. In Computer Vision–ECCV 2014 ,
pages 818–833. Springer, 2014. 2

to appear in CVPR (2015)


convolutional networks. In Computer Vision–ECCV 2014 ,
pages 818–833. Springer, 2014. 2
[39] N. Zhang, J. Donahue, R. Girshick, and T. Darrell. Partbased r-cnns for ﬁne-grained category detection. In Computer Vision–ECCV 2014 , pages 834–849. Springer, 2014.
1

to appear in CVPR (2015)


Deep Filter Banks for Texture Recognition and Segmentation
Mircea Cimpoi
University of Oxford
mircea@robots.ox.ac.ukSubhransu Maji
University of Massachusetts, Amherst
smaji@cs.umass.eduAndrea Vedaldi
University of Oxford
vedaldi@robots.ox.ac.uk
Abstract
Research in texture recognition often concentrates on the
problem of material recognition in uncluttered conditions,
an assumption rarely met by applications. In this work we
conduct a ﬁrst study of material and describable texture attributes recognition in clutter, using a new dataset derived
from the OpenSurface texture repository. Motivated by the
challenge posed by this problem, we propose a new texture
descriptor, FV-CNN, obtained by Fisher Vector pooling of a
Convolutional Neural Network (CNN) ﬁlter bank. FV-CNN
substantially improves the state-of-the-art in texture, material and scene recognition. Our approach achieves 79.8%
accuracy on Flickr material dataset and 81% accuracy on
MIT indoor scenes, providing absolute gains of more than
10% over existing approaches. FV-CNN easily transfers
across domains without requiring feature adaptation as for
methods that build on the fully-connected layers of CNNs.

Deep convolutional filter banks for texture recognition and segmentation

Summary:
Research in texture recognition often concentrates on the problem of material
recognition in uncluttered conditions, an assumption rarely met by
applications. In this work we conduct a first study of material and describable
texture at- tributes recognition in clutter, using a new dataset derived from
the OpenSurface texture repository. Motivated by the challenge posed by this
problem, we propose a new texture descriptor, D-CNN, obtained by Fisher Vector
pooling of a Convolutional Neural Network (CNN) filter bank. D-CNN
substantially improves the state-of-the-art in texture, mate- rial and scene
recognition. Our approach achieves 82.3% accuracy on Flickr material dataset
and 81.1% accuracy on MIT indoor scenes, providing absolute gains of more than
10% over existing approaches. D-CNN easily trans- fers across domains without
requiring feature adaptation as for methods that build on the fully-connected
layers of CNNs. Furthermore, D-CNN can seamlessly incorporate multi-scale
information and describe regions of arbitrary shapes and sizes. Our approach is
particularly suited at lo- calizing stuff categories and obtains
state-of-the-art re- sults on MSRC segmentation dataset, as well as promising
results on recognizing materials and surface attributes in clutter on the
OpenSurfaces dataset.

Mircea Cimpoi
Subhransu Maji
Andrea Vedaldi

Accepted to CVPR15


10% over existing approaches. FV-CNN easily transfers
across domains without requiring feature adaptation as for
methods that build on the fully-connected layers of CNNs.
Furthermore, FV-CNN can seamlessly incorporate multiscale information and describe regions of arbitrary shapes
and sizes. Our approach is particularly suited at localizing “stuff” categories and obtains state-of-the-art results
on MSRC segmentation dataset, as well as promising results
on recognizing materials and surface attributes in clutter on
the OpenSurfaces dataset.
1. Introduction
Texture is ubiquitous and provides useful cues of material properties of objects and their identity, especially when
shape is not useful. Hence, a signiﬁcant amount of effort in
the computer vision community has gone into recognizing
texture via tasks such as texture perception [1, 2, 12, 13]
and description [7, 11], material recognition [26, 36, 37],
segmentation [20, 29], and even synthesis [9, 43].
Perhaps the most studied task in texture understanding
is the one of material recognition, as captured in benchmarks such as CuRET [8], KTH-TIPS [5], and, more recently, FMD [38]. However, while at least the FMD dataset
contains images collected from the Internet, vividly dubbed

Accepted to CVPR15


contains images collected from the Internet, vividly dubbed
“images in the wild”, all these datasets make the simplifying
banded
 blotchy
 chequered
 grid
marbled
 paisley
 paisley
 wrinkled
brick
 ceramic
 carpet
 fabric
Figure 1. Texture recognition in clutter . Example of top retrieved texture segments by attributes (top two rows) and materials
(bottom) in the OpenSurfaces dataset.
assumption that textures ﬁll images. Thus, they are not necessarily representative of the signiﬁcantly harder problem
of recognising materials in natural images, where textures
appear in clutter. Building on a recent dataset collected by
the computer graphics community, the ﬁrst contribution of
this paper is a large-scale analysis of material and perceptual texture attribute recognition and segmentation in clutter(Fig. 1 and Sect. 2).
Motivated by the challenge posed by recognising texture
in clutter, we develop a new texture descriptor. In the simplest terms a texture is characterized by the arrangement of
local patterns, as captured in early works [26, 41] by the
distribution of local “ﬁlter bank” responses. These ﬁlter
banks were designed to capture edges, spots and bars at
different scales and orientations. Typical combinations of

Accepted to CVPR15


banks were designed to capture edges, spots and bars at
different scales and orientations. Typical combinations of
the ﬁlter responses, identiﬁed by vector quantisation, were
used as the computational basis of the “textons” proposed
by Julesz [22]. Texton distributions were the early versions
of “bag-of-words” representations, a dominant approach in
recognition in the early 2000s, since then improved by new
pooling schemes such as soft-assignment [27, 42, 48] and
Fisher Vectors (FVs) [32]. Until recently, FV with SIFT
features [28] as a local representation was the state-of-theart method for recognition, not only for textures, but for
objects and scenes too.
Later, however, Convolutional Neural Networks (CNNs)
1arXiv:1411.6836v2  [cs.CV]  9 Jul 2015
have emerged as the new state-of-the-art for recognition, exempliﬁed by remarkable results in image classiﬁcation [23],
detection [14] and segmentation [16] on a number of widely
used benchmarks. Key to their success is the ability to leverage large labelled datasets to learn increasingly complex

Accepted to CVPR15


detection [14] and segmentation [16] on a number of widely
used benchmarks. Key to their success is the ability to leverage large labelled datasets to learn increasingly complex
transformations of the input to capture invariances. Importantly, CNNs pre-trained on such large datasets have been
shown [6, 14, 30] to contain general-purpose feature extractors, transferrable to many other domains.
Domain transfer in CNNs is usually achieved by using
as features the output of a deep, fully-connected layer of
the network. From the perspective of textures, however,
this choice has three drawbacks. The ﬁrst one (I) is that,
while the convolutional layers are akin to non-linear ﬁlter
banks, the fully connected layers capture their spatial layout. While this may be useful for representing the shape of
an object, it may not be as useful for representing texture. A
second drawback (II) is that the input to the CNN has to be
of ﬁxed size to be compatible with the fully connected layers, which requires an expensive resizing of the input image,
particularly when features are computed for many different
regions [14, 15]. A third and more subtle drawback (III) is
that deeper layers may be more domain-speciﬁc and therefore potentially less transferrable than shallower layers.

Accepted to CVPR15


that deeper layers may be more domain-speciﬁc and therefore potentially less transferrable than shallower layers.
The second contribution of this paper is FV-CNN
(Sect. 3), a pooling method that overcomes these drawbacks . The idea is to regard the convolutional layers of a
CNN as a ﬁlter bank and build an orderless representation
using FV as a pooling mechanism, as is commonly done
in the bag-of-words approaches. Although the suggested
change is simple, the approach is remarkably ﬂexible and
effective. First, pooling is orderless and multi-scale, hence
suitable for textures. Second, any image size can be processed by convolutional layers, avoiding costly resizing operations. Third, convolutional ﬁlters, pooled by FV-CNN,
are shown to transfer more easily than fully-connected ones
even without ﬁne-tuning. While other authors [15, 18] have
recently proposed alternative pooling strategies for CNNs,
we show that our method is more natural, faster and often
signiﬁcantly more accurate.
Thethird contribution of the paper is a thorough evaluation of these descriptors on a variety of benchmarks, from
textures to objects (Sect. 4). In textures, we evaluate material and describable attributes recognition and segmentation

Accepted to CVPR15


textures to objects (Sect. 4). In textures, we evaluate material and describable attributes recognition and segmentation
on new datasets derived from OpenSurfaces (Sect. 2). When
used with linear SVMs, FV-CNN improves the state of the
art on texture recognition by a signiﬁcant margin. Like textures, scenes are also weakly structured and a bag-of-words
representation is effective. FV-CNN obtains 81.1% accuracy on the MIT indoor scenes dataset [34], signiﬁcantly
outperforming the current state-of-the-art of 70.8% [47].
What is remarkable is that, where [47] ﬁnds that CNNs
trained on scene recognition data perform better than CNNs
trained on an object domain (ImageNet), when used in FV-CNN not only is there an overall performance improvement, but the domain-speciﬁc advantage is entirely removed
(Tab. 3). This indicates that FV-CNN are in fact better at domain transfer. Our method also matches the previous best in
PASCAL VOC 2007 classiﬁcation dataset providing measurable boost over CNNs and closely approaches competitor methods on CUB 2010-2011 datasets when ground-truth
object bounding boxes are given.
FV-CNN can be used for describing regions by simply

Accepted to CVPR15


object bounding boxes are given.
FV-CNN can be used for describing regions by simply
pooling across pixels within the region. Combined with
a low-level segmentation algorithm this sufﬁces to localize textures within images. This approach is similar to a
recently proposed method called “R-CNN” for localizing
objects [14]. However, in contrast to it we do not need repeated evaluations of the CNN since the convolutional features can be computed just once for the entire image and
pooled differently. This makes FV-CNN not only faster,
but also as experiments suggest, much more accurate at
texture localization. We achieve state of the art results on
the MSRC segmentation dataset using a simple scheme of
classifying “superpixels” obtaining an accuracy of 87.0%
(previous best 86.5%). The corresponding R-CNN obtains
57.7% accuracy. Segmentation results are promising in the
OpenSurfaces dataset [4] as well.
Finally, we analyze the utility of different network layers
and architectures as ﬁlter banks, concluding that: SIFT is
competitive only with the ﬁrst few layers of a CNN (Fig. 4)

Accepted to CVPR15


and architectures as ﬁlter banks, concluding that: SIFT is
competitive only with the ﬁrst few layers of a CNN (Fig. 4)
and that signiﬁcant improvement to the underlying CNN architecture, such as the ones achieved by the very deep models of Simonyan and Zisserman [39], directly translate into
much better ﬁlter banks for texture recognition.
2. Texture recognition in clutter
A contribution of this work is the analysis of materials
and texture attributes in realistic imaging conditions. Earlier datasets such as KTH-TIPS were collected in controlled
conditions, which makes their applicability to natural images unclear. More recent datasets such as FMD and DTD
remove this limitation by building on images downloaded
from the Internet, dubbed images “in the wild”. However,
in these datasets texture always ﬁll the ﬁeld of view of the
camera. In this paper we remove this limitation by experimenting for the ﬁrst time with a large dataset of textures
collected in the wild and in cluttered conditions.
In particular, we build on the Open Surfaces (OS) dataset
that was recently introduced by Bell et al. [4] in computer
graphics. OS comprises 25,357 images, each containing a
number of high-quality texture/material segments. Many of

Accepted to CVPR15


graphics. OS comprises 25,357 images, each containing a
number of high-quality texture/material segments. Many of
these segments are annotated with additional attributes such
as the material name, the viewpoint, the BRDF, and the object class. Not all segments have a complete set of annotations; the experiments in this paper focus on the 58,928 that
contain material names. Since material classes are highly
unbalanced, only the materials that contain at least 400 examples are considered. This result in 53,915 annotated
material segments in 10,422 images spanning 22 different
classes.1Images are split evenly into training, validation,
and test subsets with 3,474 images each. Segment sizes are
highly variable, with half of them being relatively small,
with an area smaller than 6464pixels. While the lack
of exhaustive annotations makes it impossible to deﬁne a
complete background class, several less common materials
(including for example segments that annotators could not
assign to a material) are merged in an “other” class that acts
as pseudo-background.
In order to study perceptual properties as well as materials, we augment the OS dataset with some of the 47
attributes from the DTD [7]. Since the OS segments do
not trigger with sufﬁcient frequency all the 47 attributes,

Accepted to CVPR15


attributes from the DTD [7]. Since the OS segments do
not trigger with sufﬁcient frequency all the 47 attributes,
the evaluation is restricted to eleven of them for which it
was possible to identify at least 100 matching segments.2
The attributes were manually labelled in the 53,915 segments retained for materials. We refer to this data as
OSA. The complete list of images, segments, labels, and
splits are available at http://www.robots.ox.ac.
uk/˜vgg/data/wildtex/ .
3. Method
This section describes the methodological contributions
of this paper: region description and segmentation.
3.1. Region description
This section introduces a number of visual descriptors
suitable to model the appearance of image regions. Texture
is traditionally described by orderless pooling of ﬁlter bank
responses as, unlike in objects, the overall shape information is usually unimportant. However, small under-sampled
textures may beneﬁt if recognized in the context of an object. Thus, the primacy of orderless pooling may not always
hold in the recognition of textures in natural conditions.
In order to explore the interplay between shape and orderless pooling, we evaluate two corresponding region descriptors: FC-CNN for shape and FV-CNN for texture.

Accepted to CVPR15


In order to explore the interplay between shape and orderless pooling, we evaluate two corresponding region descriptors: FC-CNN for shape and FV-CNN for texture.
Both descriptors are based on the same CNN features [23]
obtained from an off-the-shelf CNN pre-trained on the ImageNet ILSVRC 2012 data as suggested in [6, 21, 35]. Since
the underlying CNN is the same, it is meaningful to compare FC- and FV-CNN directly.
1The classes and corresponding number of example segments are:
brick (610), cardboard (423), carpet/rug (1,975), ceramic (1,643), concrete (567), fabric/cloth (7,484), food (1,461), glass (4,571), granite/marble (1,596), hair (443), other (2,035), laminate (510), leather (957),
metal (4,941), painted (7,870), paper/tissue (1,226), plastic/clear (586),
plastic/opaque (1,800), stone (417), tile (3,085), wallpaper (483), wood
(9,232).
2These are: banded, blotchy, chequered, ﬂecked, gauzy, grid, marbled,

Accepted to CVPR15


(9,232).
2These are: banded, blotchy, chequered, ﬂecked, gauzy, grid, marbled,
paisley, pleated, stratiﬁed, wrinkled.Object descriptor: FC-CNN. The FC-CNN descriptor is
obtained by extracting as features the output of the penultimate Fully-Connected (FC) layer of a CNN, including
the non-linear gating function, applied to the input image.
This can be considered an object descriptor because the
fully connected layers allow FC-CNN to capture the overall
shape of the object contained in the region. FC-CNN is applied to an image region R(which may be the whole image)
by warping the bounding box enclosing R(plus a 10% border) to a square of a ﬁxed size matching the default CNN
input geometry, obtaining the same R-CNN descriptor introduced by Girshick et al. [14] as a state-of-the-art object
detector in the PASCAL VOC [10] data.
Texture descriptor: FV-CNN. The FV-CNN descriptor
is inspired by the state-of-the-art texture descriptors of [7]
based on the Fisher Vector (FV). Differently from FC-CNN,
FV pools local features densely within the described regions

Accepted to CVPR15


based on the Fisher Vector (FV). Differently from FC-CNN,
FV pools local features densely within the described regions
removing global spatial information , and is therefore more
apt at describing textures than objects. Here FV is computed
on the output of a single (last) convolutional layer of the
CNN, but we compared features from other layers as well
(Sect 4.4). By avoiding the computation of the fully connected layers, the input image does not need to be rescaled
to a speciﬁc size; in fact, the dense convolutional features
are extracted at multiple scales and pooled into a single FV
just like for SIFT. The pooled convolutional features are extracted immediately after the last linear ﬁltering operator
and are not otherwise normalised.
The FV-CNN descriptor is related to the one proposed
by Gong et al. [15]. There VLAD pooling, which is similar to FV , is applied to FC-CNN-like descriptors extracted
from densely sampled image windows. A key difference of
FV-CNN is that dense features are extracted from the convolutional rather than fully-connected layers. This is more
natural, signiﬁcantly more efﬁcient (as it does not require

Accepted to CVPR15


natural, signiﬁcantly more efﬁcient (as it does not require
recomputing the network for each extracted local descriptor) and, as shown in Sect. 4, more accurate.
3.2. Region segmentation
This section discusses our method to automatically partition an image into a number of recognisable regions. Inspired by Cimpoi et al. [7] that successfully ported object
description methods to texture descriptors, here we propose
a segmentation technique inspired by object detection. An
increasingly popular method for object detection, followed
for example by R-CNN [14], is to ﬁrst propose a number of
candidate object regions using low-level image cues, and
then verifying a shortlist of such regions using a powerful classiﬁer. Applied to textures, this requires a low-level
mechanism to generate textured region proposals, followed
by a region classiﬁer. A key advantage of this approach
is that it allows applying object- (FC-CNN) and texturelike (FV-CNN) descriptors alike. After proposal classiﬁcation, each pixel can be assigned more than one label; this is
solved with simple voting schemes, also inspired by object
detections methods.
The paper explores two such region generation methods:

Accepted to CVPR15


solved with simple voting schemes, also inspired by object
detections methods.
The paper explores two such region generation methods:
the crisp regions of [19] and the multi-scale combinatorial
grouping of [3]. In both cases, region proposals are generated using low-level image cues, such as colour or texture
consistency, as speciﬁed by the original methods. It would
of course be possible to incorporate FC-CNN and FV-CNN
among these energy terms to potentially strengthen the region generation mechanism itself. However, this contradicts partially the logic of the scheme, which breaks down
the problem into cheaply generating tentative segmentations and then verifying them using a more powerful (and
likely expensive) model. Furthermore, and more importantly, these cues focus on separating texture instances , as
presented in each particular image, whereas FC-CNN and
FV-CNN are meant to identify a texture class. It is reasonable to expect instance-speciﬁc cues (say the colour of a
painted wall) to be better for segmentation.
4. Results
This section evaluates the proposed region recognition
methods for classifying and segmenting materials, describable texture properties, and higher-level object categories.
Sect. 4.1 evaluates the classiﬁcation task by assessing how
well regions can be classiﬁed given that their true extent is

Accepted to CVPR15


Sect. 4.1 evaluates the classiﬁcation task by assessing how
well regions can be classiﬁed given that their true extent is
known and Sect. 4.3 evaluates both classiﬁcation and segmentation . The rest of the section introduces the evaluation
benchmarks and technical details of the representations.
Datasets. The evaluation considers three texture recognition benchmarks other than OS (Sect. 2). The ﬁrst one is the
Flickr Material Dataset (FMD) [37], a recent benchmark
containing 10 material classes. The second one is the Describable Texture Datasets (DTD) [7], which contains texture images jointly annotated with 47 describable attributes
drawn from the psychological literature. Both FMD and
DTD contain images “in the wild”, i.e. collected in uncontrolled conditions. However, differently from OS, these images are uncluttered. The third texture dataset is KTH-TIPS2b[5, 17], containing a number of example images for each
of four samples of 11 material categories. For each material, images of one sample are used for training and the
remaining for testing.
Object categorisation is evaluated in the PASCAL VOC
2007 [10] dataset, containing 20 object categories, any
combination of which may be associated to any of the

Accepted to CVPR15


Object categorisation is evaluated in the PASCAL VOC
2007 [10] dataset, containing 20 object categories, any
combination of which may be associated to any of the
benchmark images. Scene categorisation uses the MIT Indoor [34] dataset, containing 67 indoor scene classes. Finegrained categorisation uses the Caltech/UCSD Bird dataset
(CUB) [45], containing images of 200 bird species.
Note that some of these datasets come with ground truth
region/object localisation. The +R sufﬁx will be appendedto a dataset to indicate that this information is used both at
training and testing time. For example, OS means that segmentation is performed automatically at test time, whereas
OS+R means that ground-truth segmentations are used.
Evaluation measures. For each dataset the corresponding standard evaluator protocols and accuracy measures are
used. In particular, for FMD, DTD, MIT Indoor, CUB,
and OS+R, evaluation uses average classiﬁcation accuracy,
per-segment/image and normalized for each class. When
evaluating the quality of a segmentation algorithm, however, one must account for the fact that in most datasets,
and in OS and MSRC in particular, not all pixels are labelled. In this case, accuracy is measured per-pixel rather

Accepted to CVPR15