and in OS and MSRC in particular, not all pixels are labelled. In this case, accuracy is measured per-pixel rather
than per-segment, ignoring all pixels that are unlabelled in
the ground truth. For MSRC, furthermore, accuracy is normalised across all pixels regardless of their category. For
OSA, since some segments may have more than one label,
we are reporting mAP, following the standard procedure for
multi-label datasets. Finally, PASCAL VOC 2007 classiﬁcation uses mean average precision (mAP), computed using
the TRECVID 11-point interpolation [10].3
Descriptor details. FC-CNN and FV-CNN build on the
pre-trained VGG-M [6] model as this performs better than
other popular models such as [21] while having a similar computational cost. This network results in 4096dimensional FC-CNN features and 512-dimensional local
features for FV-CNN computation. The latter are pooled
into a FV representation with 64 Gaussian components,
resulting in 65K-dimensional descriptors. While the FVCNN dimensionality is much higher than the 4K dimensions of FC-CNN, the FV is known to be highly redundant and can be typically compressed by one order of magnitude without appreciable reduction in the classiﬁcation

Accepted to CVPR15


performance [31], so the effective dimensionality of FCand FV-CNN is likely comparable. We veriﬁed that by
PCA-reducing FV to 4096 dimensions and observing only
a marginal reduction in classiﬁcation performance in the
PASCAL VOC object recognition task described below. In
addition to VGG-M, the recent state-of-the art VGG-VD
(very deep with 19 layers) model of Simonyan and Zisserman [39] is also evaluated.
Due to the similarity between FV-CNN and the dense
SIFT FV descriptors used for texture recognition in [7],
the latter is evaluated as well. Since SIFT descriptors
are smaller (128-dimensional) than the convolutional ones
(512-dimensional), a larger number of Gaussian components (256) are used to obtain FV descriptors with a comparable dimensionality. The SIFT descriptors support is
3232pixels at the base scale.
In order to make results comparable to [7], we use
the same settings whenever possible. FV-CNN and DSIFT compute features after rescaling the image by factors
3The deﬁnition of AP was changed in later versions of the benchmark.

Deep convolutional filter banks for texture recognition and segmentation

Summary:
Research in texture recognition often concentrates on the problem of material
recognition in uncluttered conditions, an assumption rarely met by
applications. In this work we conduct a first study of material and describable
texture at- tributes recognition in clutter, using a new dataset derived from
the OpenSurface texture repository. Motivated by the challenge posed by this
problem, we propose a new texture descriptor, D-CNN, obtained by Fisher Vector
pooling of a Convolutional Neural Network (CNN) filter bank. D-CNN
substantially improves the state-of-the-art in texture, mate- rial and scene
recognition. Our approach achieves 82.3% accuracy on Flickr material dataset
and 81.1% accuracy on MIT indoor scenes, providing absolute gains of more than
10% over existing approaches. D-CNN easily trans- fers across domains without
requiring feature adaptation as for methods that build on the fully-connected
layers of CNNs. Furthermore, D-CNN can seamlessly incorporate multi-scale
information and describe regions of arbitrary shapes and sizes. Our approach is
particularly suited at lo- calizing stuff categories and obtains
state-of-the-art re- sults on MSRC segmentation dataset, as well as promising
results on recognizing materials and surface attributes in clutter on the
OpenSurfaces dataset.

Mircea Cimpoi
Subhransu Maji
Andrea Vedaldi

Accepted to CVPR15


Accepted to CVPR15


Table 1. Evaluation of texture descriptors. The table compares FC-CNN, FV-CNN on two networks trained on ImageNet – VGG-M and
VGG-VD, and IFV on dense SIFT. We evaluated these descriptors on (a) material datasets (FMD, KTH-T2b, OS+R), (b) texture attributes
(DTD, OSA+R) and (c) general categorisation datasets (MSRC+R,VOC07,MIT Indoor) and ﬁne grained categorisation (CUB, CUB+R).
For this experiment the region support is assumed to be known (and equal to the entire image for all the datasets except OS+R and MSRC+R
and for CUB+R, where it is set to the bounding box of a bird). () using a model without parts. Best results are marked in bold.
VGG-M VGG-VD
dataset measure (%) FC-CNN FV-CNN FV+FC-CNN FC-CNN FV-CNN FC+FV-CNN SoA
OS pp-acc 36.3 48.7 (46.9) 50.5 38.8 55.4 (55.7) 55.2 –

Accepted to CVPR15


OS pp-acc 36.3 48.7 (46.9) 50.5 38.8 55.4 (55.7) 55.2 –
MSRC msrc-acc 56.1 82.3 75.5 57.7 87.0 80.4 86.5 [24]
Table 2. Segmentation and recognition using crisp region proposals of materials (OS) and things & stuff (MSRC). Per-pixel accuracies are
reported, using the MSRC variant (see text) for the MSRC dataset. Results using MCG proposals [3] are reported in brackets for FV-CNN.
2s; s=3;2:5; : : :1:5(but, for efﬁciency, discarding
scales that would make the image larger than 10242pixels). Before pooling descriptors with a FV , these are usually
de-correlated by using PCA. Here PCA is applied to SIFT,
additionally reducing its dimension to 80, as this was empirically shown to improve the overall recognition performance. However, PCA is not applied to the convolutional
features in FV-CNN as in this case results were worse.
Learning details. The region descriptors (FC-CNN, FVCNN, and D-SIFT) are classiﬁed using 1-vs-rest Support

Accepted to CVPR15


Learning details. The region descriptors (FC-CNN, FVCNN, and D-SIFT) are classiﬁed using 1-vs-rest Support
Vector Machine (SVM) classiﬁers. Prior to learning, descriptors are L2normalised and the learning constant set
toC= 1. This is motivated by the fact that, after data
normalisation, the exact choice of Chas a negligible effect
on performance. Furthermore, the accuracy of the 1-vs-rest
classiﬁcation scheme is improved by recalibrating the SVM
scores after training, by scaling the SVM weight vector and
bias such that the median scores of the negative and positive
training samples for each class are mapped respectively to
the values1and1.
4.1. Region recognition: textures
This and the following section evaluate region recognition assuming that the ground-truth region Ris known (Table 1), for example because it ﬁlls the entire image. This
section focuses on textures (materials and perceptual attributes), while the next on objects and scenes.
Texture recognition without clutter. This experimentevaluates the performance of FC-CNN, FV-CNN, D-SIFT,
and their combinations in standard texture recognition

Accepted to CVPR15


Texture recognition without clutter. This experimentevaluates the performance of FC-CNN, FV-CNN, D-SIFT,
and their combinations in standard texture recognition
benchmarks such as FMD, KTH-TIPS-2, and DTD. FCCNN is roughly equivalent to the DeCAF method used
in [7] for this data as regions ﬁll images; however, while the
performance of our FC-CNN is similar in KTH ( 70%),
it is substantially better in FMD ( 60:7%!70:4%accuracy) and DTD ( 54:8%!58:7%). This likely is caused by
the improved underlying CNN, an advantage which is more
obvious in FMD and DTD that are closer to object recognition than KTH. FV-CNN performs within 2%in FMD and
KTH but substantially better for DTD ( 58:7%!66:6%).
D-SIFT is comparable in performance to FC-CNN in DTD
and KTH, but substantially worse ( 70:4%!59:2%) in
FMD. Our conclusion is that, even when textures ﬁll the
input image as in these benchmarks, orderless pooling in
FV-CNN and D-SIFT can be either the same or substantially better than the pooling operated in the fully-connected

Accepted to CVPR15


input image as in these benchmarks, orderless pooling in
FV-CNN and D-SIFT can be either the same or substantially better than the pooling operated in the fully-connected
layers by FC-CNN.
Combining FC- and FV-CNN improves performance in
all datasets by 13%. While this combination is already signiﬁcantly above the state-of-the-art in DTD and
FMD ( +2:6%=11:2%), the method of [7] still outperforms
these descriptors in KTH. However, replacing VGG-M with
VGG-VD signiﬁcantly improves the performance in all
cases – a testament to the power of deep features. In particular, the best method FC+FV-CNN-VD, improves the state
of the art by at least 6%in all datasets. Interestingly, this is
obtained by using a single low-level feature type as FC- and
FV-CNN build on the same convolutional features. Adding
D-SIFT results in at most 1%improvement, and in some
cases it slightly degrades performance.
Texture recognition in clutter. The advantage of FV-CNN
over FC-CNN is much larger when textures do not ﬁll the

Accepted to CVPR15


cases it slightly degrades performance.
Texture recognition in clutter. The advantage of FV-CNN
over FC-CNN is much larger when textures do not ﬁll the
image but are extracted from clutter. In OS+R (Sect. 2),
material recognition accuracy starts at about 46% for both
FC-CNN and D-SIFT; however, FV-CNN improves this by
more than 11% (46:5%!58:1%). The combination of FCand FV-CNN improves results further by 2%, but adding
SIFT deteriorates performance. With the very deep CNN
conclusions are similar; however, switching to VGG-VD
barely affects the FC-CNN performance ( 46:5!48:0%),
but strongly affects the one of FV-CNN ( 58:1%!65:1%).
This conﬁrms that FC-CNN, while excellent in object detection, is not a very good descriptor for classifying textured
regions. Results in OSA+R for texture attribute recognition
(Sect. 2) and in MSRC+R for semantic segmentation are
analogous; it is worth noting that, when ground-truth segments are used in this experiment, the best model achieves
a nearly perfect 99:7%classiﬁcation rate in MSRC.

Accepted to CVPR15


a nearly perfect 99:7%classiﬁcation rate in MSRC.
4.2. Region recognition: objects and scenes
This section shows that the FV-CNN descriptor, despite
its orderless nature that make it an excellent texture descriptor, excels at object and scene recognition as well. In the remainder of the section, and unless otherwise noted, region
descriptors are applied to images as a whole by considering
these single regions.
FV-CNN vs FC-CNN. As seen in Table 1, in PASCAL
VOC and MIT Indoor the FC-CNN descriptor performs
very well but in line with previous results for this class of
methods [6]. FV-CNN performs similarly to FC-CNN in
PASCAL VOC, but substantially better ( +5% for VGG-M
and +13% for VGG-VD) in MIT Indoor. As further discussed below, this is an example of the ability of FV-CNN
to transfer between domains better than FC-CNN. The gap
between FC-CNN and FV-CNN is the highest for the very
deep VGG-VD models (68.1% !81.1%), a trend also exhibited by other datasets as seen in Tab. 1. In the CUB
dataset, FV-CNN signiﬁcantly outperforms FC-CNN both

Accepted to CVPR15


dataset, FV-CNN signiﬁcantly outperforms FC-CNN both
whether the descriptor is computed from the whole image
(CUB) or from the bird bounding box (CUB+R). In the latter case, the difference is very large ( +1014%).
Comparison with alternative pooling methods. FV-CNN
is related to the method of [15], which uses a similar
underlying CNN and VLAD instead of FV for pooling.
Notably, however, FV-CNN results on MIT Indoor are
markedly better than theirs for both VGG-M and VGGVD ( 68:8%!73:5%=81:1%resp.) and marginally better (68:8%!69:1%) when the same CAFFE CNN is used
(Tab. 3). The key difference is that FV-CNN pools convolu-Accuracy (%)
CNN FC-CNN FV-CNN FC+FV-CNN
PLACES 65.0 67.6 73.1
CAFFE 58.6 69.7 71.6
VGG-M 62.5 74.2 74.4
VGG-VD 67.6 81.0 80.3
Table 3. Accuracy of various CNNs on the MIT indoor dataset.

Accepted to CVPR15


VGG-VD 67.6 81.0 80.3
Table 3. Accuracy of various CNNs on the MIT indoor dataset.
tional features, whereas [15] pools fully-connected descriptors extracted from square image patches. Thus, even without spatial information as used by [15], FV-CNN is not only
substantially faster, but at least as accurate. Using the same
settings, that is, the same net and the same three scales, our
approach results in an 8.5 speedup.
Comparison with the state-of-the-art. The best result obtained in PASCAL VOC is comparable to the current stateof-the-art set by the deep learning method of [44] ( 85:2%!
85:0%), but using a much more straightforward pipeline. In
MIT Places our best performance is also substantially superior ( +10% ) to the current state-of-the-art using deep convolutional networks learned on the MIT Place dataset [47]
(see also below). In the CUB dataset, our best performance
is a little short (3%) of the state-of-the-art results of [46].
However, [46] uses a category-speciﬁc part detector and
corresponding part descriptor as well as a CNN ﬁne-tuned

Accepted to CVPR15


However, [46] uses a category-speciﬁc part detector and
corresponding part descriptor as well as a CNN ﬁne-tuned
on the CUB data; by contrast, FV-CNN and FC-CNN are
used here as global image descriptors which, furthermore,
are the same for all the datasets considered . Compared to
the results of [46] without part-based descriptors (but still
using a part-based object detector), our global image descriptors perform substantially better ( 62:1%!69:1%).
We conclude that FV-CNN is a very strong descriptor.
Results are usually as good, and often signiﬁcantly better,
than FC-CNN. In most applications, furthermore, FV-CNN
is many times faster as it does not require evaluating the
CNN for each target image region. Finally, FC- and FVCNN can be combined outperforming the state-of-the-art in
many benchmarks.
Domain transfer. So far, the same underlying CNN features, trained on ImageNet’s ILSVCR, were used for all
datasets. Here we investigate the effect of using domainspeciﬁc features. To do so, we consider the PLACES [47],

Accepted to CVPR15


datasets. Here we investigate the effect of using domainspeciﬁc features. To do so, we consider the PLACES [47],
trained to recognize places on a dataset of about 2.5 million labeled images. [47] showed that, applied to the task of
scene recognition in MIT Indoor, these features outperform
similar ones trained on ILSVCR (denoted CAFFE [21] below) – a fact explained by the similarity of domains. Below,
we repeat this experiment using FC- and FV-CNN descriptors on top of VGG-M, VGG-VD, PLACES, and CAFFE.
Results are shown in Table 3. The FC-CNN results are
in line with those reported in [47] – in scene recognition
with FC-CNN the same CNN architecture performs better
if trained on the Places dataset instead of the ImageNet data
(58:6%!65:0%accuracy4). Nevertheless, stronger CNN
architectures such as VGG-M and VGG-VD can approach
and outperform PLACES even if trained on ImageNet data
(65:0%!63:0%=68:1%).
However, when it comes to using the ﬁlter banks with
FV-CNN, conclusions are very different. First, FV-CNN

Accepted to CVPR15


However, when it comes to using the ﬁlter banks with
FV-CNN, conclusions are very different. First, FV-CNN
outperforms FC-CNN in all cases, with substantial gains
up to 20% in correspondence of a domain transfer from
ImageNet to MIT Indoor. Second, the advantage of using domain-speciﬁc CNNs disappears . In fact, the same
CAFFE model that is 6.4% worse than PLACES with FCCNN, is actually 1.5% better when used in FV-CNN. The
conclusion is that FV-CNN appears to be immune, or at
least substantially less sensitive, to domain shifts.
Our tentative explanation of this surprising phenomenon
is that the convolutional layers are less committed to a speciﬁc dataset than the fully ones. Hence, by using those,
FV-CNN tends to be a more general than FC-CNN.
4.3. Texture segmentation
The previous section considered the problem of region
recognition when the region support is known at test time.
This section studies the problem of recognising regions
when their extent Risnotknown and also be estimated.
The ﬁrst experiment (Tab. 2) investigates the simplest possible scheme: combining the region descriptors
of Sect. 4.1 with a general-purpose image segmentation

Accepted to CVPR15


The ﬁrst experiment (Tab. 2) investigates the simplest possible scheme: combining the region descriptors
of Sect. 4.1 with a general-purpose image segmentation
method, namely the crisp regions of [19]. Two datasets
are evaluated: OS for material recognition and MSRC for
things & stuff. Compared to OS+R, classifying crisp regions results in a drop of about 5%points for all descriptors. As this dataset is fairly challenging with best achievable performance is 55:4%, this is a satisfactory result. But
it also illustrates that there is ample space for future improvements. In MSRC, the best accuracy is 87:0%, just a
hair above the best published result 86:5%[25]. Remarkably, these algorithms not use any dataset-speciﬁc training, nor CRF-regularised semantic inference: they simply
greedily classify regions as obtained from a general-purpose
segmentation algorithms. Qualitative segmentation results
(sampled at random) are given in Fig. 2 and 3.
Unlike crisp regions, the proposals of [3] are overlapping and a typical image contains thousands of them. We
propose a simple scheme to combine prediction from multiple proposals. For each proposal we set its label to the
highest scoring class, and score to the highest score. We
then sort the proposals in the increasing order of their score

Accepted to CVPR15


highest scoring class, and score to the highest score. We
then sort the proposals in the increasing order of their score
divided by their area and paste them one by one. This has
the effect of considering larger regions before smaller ones
and more conﬁdent regions after less ones for regions of
the same area. Results using FV-CNN shown in Tab. 2 in
4[47] report 68:3%for PLACES applied to MIT Indoor, a small difference explained by implementation details such as the fact that, for all the
methods, we do not perform data augmentation by jittering.
40444852566064687276Performance (%)
conv1
[7 x  7]
 96x256
conv2
[27 x  27]
256x128
conv3
[75 x  75]
512x 64
conv4
[107 x  107]
512x 64
conv5
[139 x  139]
512x 64(VGG−M) Filterbank Analysis

VOC07 (mAP11)
MIT Indoor (acc)
FMD (acc)
DTD (acc)
SIFTFigure 4. CNN ﬁlterbank analysis for VGG-M. Performance of
ﬁlter banks extracted from various layers network are shown on

Accepted to CVPR15


SIFTFigure 4. CNN ﬁlterbank analysis for VGG-M. Performance of
ﬁlter banks extracted from various layers network are shown on
various datasets. For each layer convf1; : : : ; 5gwe show, the size
of the receptive ﬁeld [ NN], and FBD, where FBis the size
of ﬁlter bank and Dis the dictionary size in the FV representation.
The performance using SIFT is shown in black plus (+) marks.
brackets (FC-CNN was too slow for our experiments). The
results are comparable to those using crisp regions, and we
obtain 55:7%accuracy on the OS dataset. Our initial attempts at schemes such as non-maximum suppression of
overlapping regions that are quite successful for object segmentation [16] performed rather poorly. We believe this is
because unlike objects, material information is fairly localized and highly irregularly shaped in an image. However,
there is room for improvement by combining evidence from
multiple segmentations.
4.4. Convolutional layer analysis
We study the performance of ﬁlter banks extracted from
different layers of a CNN in the FV-CNN framework. We

Accepted to CVPR15


4.4. Convolutional layer analysis
We study the performance of ﬁlter banks extracted from
different layers of a CNN in the FV-CNN framework. We
use the VGG-M network which has ﬁve convolutional layers. Results on various datasets, obtained as in Sect. 4.1
and 4.2, are shown in Fig. 4. In addition we also show the
performance using FVs constructed from dense SIFT using
a number of words such that the resulting FV is roughly the
same size of FV-CNN. The CNN ﬁlter banks from layer
3 and beyond signiﬁcantly outperform SIFT. The performance monotonically improves from layer one to ﬁve.
5. Conclusions
We have conducted a range of experiments on material
and texture attribute recognition in a large dataset of textures in clutter. This benchmark was derived from OpenSurfaces, an earlier contribution of the computer graphics community, highlights the potential for collaboration between
computer graphics and vision communities. We have also
evaluated a number of state-of-the-art texture descriptors on
these and many other benchmarks. Our main ﬁnding is that
orderless pooling of convolutional neural network features
is a remarkably good texture descriptor, versatile enough to
dubbed as a scene and object descriptor, resulting in new

Accepted to CVPR15


orderless pooling of convolutional neural network features
is a remarkably good texture descriptor, versatile enough to
dubbed as a scene and object descriptor, resulting in new
state-of-the-art performance in several benchmarks.
brick cardboard carpet_rug ceramic concrete fabric_cloth food glass
granite_marble hair other laminate leather metal painted paper_tissue
plastic_clear plastic_opaque stone tile wallpaper wood(a) (b) (c) (d) (e) (f)
Figure 2. OS material recognition results. Example test image with material recognition and segmentation on the OS dataset. (a) original
image. (b) ground truth segmentations from the OpenSurfaces repository (note that not all pixels are annotated). (c) FC-CNN and crispregion proposals segmentation results. (d) incorrectly predicted pixels (restricted to the ones annotated). (e-f) the same, but for FV-CNN.
building grass tree cow sheep sky aeroplane water
face car bicycle flower sign bird book chair
road cat dog body boat
(a) (b) (c) (d) (e) (f)
Figure 3. MSRC object segmentation results. (a) image, (b) ground-truth, (c-d) FC-CNN, (d-e) FV-CNN segmentation and errors.

Accepted to CVPR15


Acknowledgements M. Cimpoi was supported by ERC
grant VisRec no. 228180 and the XRCE UAC grant. S. Maji
acknowledges a faculty startup grant from UMass, Amherst.References
[1] E. H. Adelson. On seeing stuff: The perception of materials by humans and machines. SPIE , 4299, 2001. 1
[2] M. Amadasun and R. King. Textural features corresponding to textural properties. Systems, Man, and Cybernetics , 19(5), 1989. 1
[3] P. Arbel ´aez, J. Pont-Tuset, J. T. Barron, F. Marques, and J. Malik.
Multiscale combinatorial grouping. CVPR, 2014. 4, 5, 7
[4] S. Bell, P. Upchurch, N. Snavely, and K. Bala. Opensurfaces: A
richly annotated catalog of surface appearance. In Proc. SIGGRAPH ,
2013. 2
[5] B. Caputo, E. Hayman, and P. Mallikarjuna. Class-speciﬁc material
categorisation. In ICCV , 2005. 1, 4

Accepted to CVPR15


categorisation. In ICCV , 2005. 1, 4
[6] K. Chatﬁeld, K. Simonyan, A. Vedaldi, and A. Zisserman. Return
of the devil in the details: Delving deep into convolutional nets. In
Proc. BMVC , 2014. 2, 3, 4, 6
[7] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.
Describing textures in the wild. In Proc. CVPR , 2014. 1, 3, 4, 5
[8] K. J. Dana, B. van Ginneken, S. K. Nayar, and J. J. Koenderink.
Reﬂectance and texture of real world surfaces. ACM Transactions on
Graphics , 18(1):1–34, 1999. 1
[9] A. Efros and T. Leung. Texture synthesis by non-parametric sampling. In CVPR , volume 2, 1999. 1
[10] M. Everingham, A. Zisserman, C. Williams, and L. V . Gool. The
PASCAL visual obiect classes challenge 2007 (VOC2007) results.
Technical report, Pascal Challenge, 2007. 3, 4

Accepted to CVPR15


PASCAL visual obiect classes challenge 2007 (VOC2007) results.
Technical report, Pascal Challenge, 2007. 3, 4
[11] V . Ferrari and A. Zisserman. Learning visual attributes. In Proc.
NIPS , 2007. 1
[12] D. Forsyth. Shape from texture and integrability. In ICCV , volume 2,
pages 447–452. IEEE, 2001. 1
[13] J. G ˚arding. Shape from texture for smooth curved surfaces in perspective projection. Journal of Mathematical Imaging and Vision ,
2(4):327–350, 1992. 1
[14] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature
hierarchies for accurate object detection and semantic segmentation.
InProc. CVPR , 2014. 2, 3
[15] Y . Gong, L. Wang, R. Guo, and S. Lazebnik. Multi-scale orderless
pooling of deep convolutional activation features. In Proc. ECCV ,
2014. 2, 3, 6
[16] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simultaneous
detection and segmentation. In Computer Vision–ECCV 2014 , pages

Accepted to CVPR15


detection and segmentation. In Computer Vision–ECCV 2014 , pages
297–312. Springer, 2014. 2, 7
[17] E. Hayman, B. Caputo, M. Fritz, and J.-O. Eklundh. On the significance of real-world conditions for material classiﬁcation. ECCV ,
2004. 4
[18] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep
convolutional networks for visual recognition. In Proc. ECCV , 2014.
2
[19] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp boundary
detection using pointwise mutual information. In Proc. ECCV , 2014.
4, 7
[20] A. Jain and F. Farrokhnia. Unsupervised texture segmentation using
gabor ﬁlters. Pattern recognition , 24(12):1167–1186, 1991. 1
[21] Y . Jia. Caffe: An open source convolutional architecture for fast
feature embedding. http://caffe.berkeleyvision.org/ ,
2013. 3, 4, 6
[22] B. Julesz and J. R. Bergen. Textons, the fundamental elements in

Accepted to CVPR15


2013. 3, 4, 6
[22] B. Julesz and J. R. Bergen. Textons, the fundamental elements in
preattentive vision and perception of textures. Bell System Technical
Journal , 62(6, Pt 3):1619–1645, Jul-Aug 1983. 1
[23] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classiﬁcation with deep convolutional neural networks. In Proc. NIPS , 2012.
2, 3
[24] L. Ladicky, C. Russell, P. Kohli, and P. Torr. Graph cut based inference with co-occurrence statistics. In Proc. ECCV , pages 239–253,
2010. 5
[25] L. Ladicky, P. Sturgess, K. Alahari, C. Russell, and P. Torr. What,
where and how many? combining object detectors and crfs. In Proc.
ECCV , 2010. 7
[26] T. Leung and J. Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons. International
Journal of Computer Vision , 43(1):29–44, 2001. 1

Accepted to CVPR15


Journal of Computer Vision , 43(1):29–44, 2001. 1
[27] L. Liu, L. Wang, and X. Liu. In defense of soft-assignment coding.
InComputer Vision (ICCV), 2011 IEEE International Conference on ,
pages 2486–2493. IEEE, 2011. 1[28] D. G. Lowe. Object recognition from local scale-invariant features.
InProc. ICCV , 1999. 1
[29] B. Manjunath and R. Chellappa. Unsupervised texture segmentation
using markov random ﬁeld models. IEEE Transactions on Pattern
Analysis and Machine Intelligence , 13(5):478–482, 1991. 1
[30] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and Transferring Mid-Level Image Representations using Convolutional Neural
Networks. In Proc. CVPR , 2014. 2
[31] O. M. Parkhi, K. Simonyan, A. Vedaldi, and A. Zisserman. A compact and discriminative face track descriptor. In Proc. CVPR , 2014.
4

Accepted to CVPR15


4
[32] F. Perronnin, J. S ´anchez, and T. Mensink. Improving the ﬁsher kernel for large-scale image classiﬁcation. In Computer Vision–ECCV
2010 , pages 143–156. Springer, 2010. 1
[33] X. Qi, R. Xiao, C. G. Li, Y . Qiao, J. Guo, and X. Tang. Pairwise rotation invariant co-occurrence local binary pattern. PAMI ,
36(11):2199–2213, Nov 2014. 5
[34] A. Quattoni and A. Torralba. Recognizing indoor scenes. In Proc.
CVPR , 2009. 2, 4
[35] A. S. Razavin, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: An astounding baseline for recognition. In DeepVision workshop , 2014. 3
[36] G. Schwartz and K. Nishino. Visual material traits: Recognizing
per-pixel material context. In Proc. CVCP , 2013. 1
[37] L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson. Recognizing
materials using perceptually inspired features. International Journal

Accepted to CVPR15


[37] L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson. Recognizing
materials using perceptually inspired features. International Journal
of Computer Vision , 103(3):348–371, 2013. 1, 4, 5
[38] L. Sharan, R. Rosenholtz, and E. H. Adelson. Material perceprion:
What can you see in a brief glance? Journal of Vision , 9:784(8),
2009. 1
[39] K. Simonyan and A. Zisserman. Very deep convolutional networks
for large-scale image recognition. CoRR , abs/1409.1556, 2014. 2, 4
[40] M. Sulc and J. Matas. Fast features invariant to rotation and scale of
texture. Technical report, 2014. 5
[41] M. Varma and A. Zisserman. Texture classiﬁcation: Are ﬁlter banks
necessary? In CVPR , volume 2, pages II–691. IEEE, 2003. 1
[42] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y . Gong. Localityconstrained linear coding for image classiﬁcation. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on ,

Accepted to CVPR15


pages 3360–3367. IEEE, 2010. 1
[43] L. Wei and M. Levoy. Fast texture synthesis using treestructured vector quantization. In SIGGRAPH , pages 479–488. ACM
Press/Addison-Wesley Publishing Co., 2000. 1
[44] Y . Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y . Zhao, and S. Yan. Cnn:
Single-label to multi-label. 2014. 5, 6
[45] P. Welinder, S. Branson, T. Mita, C. Wah, and F. Schroff. Caltechucsd birds 200. Technical report, Caltech-UCSD, 2010. 4
[46] N. Zhang, J. Donahue, R. Girshickr, and T. Darrell. Part-based RCNNs for ﬁne-grained category detection. In Proc. ECCV , 2014. 5,
6
[47] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning
deep features for scene recognition using places database. In Proc.
NIPS , 2014. 2, 5, 6, 7

Accepted to CVPR15


deep features for scene recognition using places database. In Proc.
NIPS , 2014. 2, 5, 6, 7
[48] X. Zhou, K. Yu, T. Zhang, and T. S. Huang. Image classiﬁcation
using super-vector coding of local image descriptors. In Computer
Vision–ECCV 2010 , pages 141–154. Springer, 2010. 1

Accepted to CVPR15


Material Recognition in the Wild with the Materials in Context Database
Sean BellPaul UpchurchNoah Snavely Kavita Bala
Department of Computer Science, Cornell University
fsbell,paulu,snavely,kb g@cs.cornell.edu
Abstract
Recognizing materials in real-world images is a challenging task. Real-world materials have rich surface texture,
geometry, lighting conditions, and clutter, which combine
to make the problem particularly difﬁcult. In this paper, we
introduce a new, large-scale, open dataset of materials in
the wild, the Materials in Context Database (MINC ), and
combine this dataset with deep learning to achieve material
recognition and segmentation of images in the wild.
MINC is an order of magnitude larger than previous material databases, while being more diverse and well-sampled
across its 23 categories. Using MINC , we train convolutional neural networks (CNNs) for two tasks: classifying
materials from patches, and simultaneous material recognition and segmentation in full images. For patch-based classiﬁcation on MINC we found that the best performing CNN
architectures can achieve 85.2% mean class accuracy. We
convert these trained CNN classiﬁers into an efﬁcient fully

Summary:
Recognizing materials in real-world images is a challenging task. Real-world
materials have rich surface texture, geometry, lighting conditions, and
clutter, which combine to make the problem particularly difficult. In this
paper, we introduce a new, large-scale, open dataset of materials in the wild,
the Materials in Context Database (MINC), and combine this dataset with deep
learning to achieve material recognition and segmentation of images in the
wild.
  MINC is an order of magnitude larger than previous material databases, while
being more diverse and well-sampled across its 23 categories. Using MINC, we
train convolutional neural networks (CNNs) for two tasks: classifying materials
from patches, and simultaneous material recognition and segmentation in full
images. For patch-based classification on MINC we found that the best
performing CNN architectures can achieve 85.2% mean class accuracy. We convert
these trained CNN classifiers into an efficient fully convolutional framework
combined with a fully connected conditional random field (CRF) to predict the
material at every pixel in an image, achieving 73.1% mean class accuracy. Our
experiments demonstrate that having a large, well-sampled dataset such as MINC
is crucial for real-world material recognition and segmentation.

Sean Bell
Paul Upchurch
Noah Snavely
Kavita Bala

CVPR 2015. Sean Bell and Paul Upchurch contributed equally


architectures can achieve 85.2% mean class accuracy. We
convert these trained CNN classiﬁers into an efﬁcient fully
convolutional framework combined with a fully connected
conditional random ﬁeld (CRF) to predict the material at
every pixel in an image, achieving 73.1% mean class accuracy. Our experiments demonstrate that having a large,
well-sampled dataset such as MINC is crucial for real-world
material recognition and segmentation.
1. Introduction
Material recognition plays a critical role in our understanding of and interactions with the world. To tell whether
a surface is easy to walk on, or what kind of grip to use
to pick up an object, we must recognize the materials that
make up our surroundings. Automatic material recognition
can be useful in a variety of applications, including robotics,
product search, and image editing for interior design. But recognizing materials in real-world images is very challenging.
Many categories of materials, such as fabric or wood, are
visually very rich and span a diverse range of appearances.
Materials can further vary in appearance due to lighting and
shape. Some categories, such as plastic and ceramic, are ofAuthors contributed equally
CNN
Sliding
CNN“wood”
Transfer weights(trained)
(discretely


CNN
Sliding
CNN“wood”
Transfer weights(trained)
(discretely
optimized)(fixed)AMT AMT AMT MINC
3 million
patches OpenSurfacesFlickr
Houzz(a) Constructing MINC
Material
Labels(b) Patch material classification
(c) Full scene material classificationP(material)
Dense
CRF
PatchFigure 1. Overview. (a) We construct a new dataset by combining
OpenSurfaces [ 1] with a novel three-stage Amazon Mechanical
Turk (AMT) pipeline. (b)We train various CNNs on patches from
MINC to predict material labels. (c)We transfer the weights to a
fully convolutional CNN to efﬁciently generate a probability map
across the image; we then use a fully connected CRF to predict the
material at every pixel.
ten smooth and featureless, requiring reasoning about subtle
cues or context to differentiate between them.
Large-scale datasets (e.g., ImageNet [ 21], SUN [ 31,19]
and Places [ 34]) combined with convolutional neural networks (CNNs) have been key to recent breakthroughs in
object recognition and scene classiﬁcation. Material recognition is similarly poised for advancement through large-scale
data and learning. To date, progress in material recognition


object recognition and scene classiﬁcation. Material recognition is similarly poised for advancement through large-scale
data and learning. To date, progress in material recognition
has been facilitated by moderate-sized datasets like the Flickr
Material Database (FMD) [ 26]. FMD contains ten material
categories, each with 100 samples drawn from Flickr photos.
These images were carefully selected to illustrate a wide
range of appearances for these categories. FMD has been
used in research on new features and learning methods for
material perception and recognition [ 17,10,20,25]. While
FMD was an important step towards material recognition, it
is not sufﬁcient for classifying materials in real-world im1arXiv:1412.0623v2  [cs.CV]  14 Apr 2015
agery. This is due to the relatively small set of categories,
the relatively small number of images per category, and also
because the dataset has been designed around hand-picked
iconic images of materials. The OpenSurfaces dataset [ 1]
addresses some of these problems by introducing 105,000
material segmentations from real-world images, and is signiﬁcantly larger than FMD. However, in OpenSurfaces many
material categories are under-sampled, with only tens of
images.


material categories are under-sampled, with only tens of
images.
A major contribution of our paper is a new, well-sampled
material dataset, called the Materials in Context Database
(MINC ), with 3 million material samples. MINC is more
diverse, has more examples in less common categories, and
is much larger than existing datasets. MINC draws data from
both Flickr images, which include many “regular” scenes,
as well as Houzz images from professional photographers of
staged interiors. These sources of images each have different
characteristics that together increase the range of materials
that can be recognized. See Figure 2 for examples of our
data. We make our full dataset available online at http:
//minc.cs.cornell.edu/.
We use this data for material recognition by training different CNN architectures on this new dataset. We perform
experiments that illustrate the effect of network architecture, image context, and training data size on subregions
(i.e., patches) of a full scene image. Further, we build on
our patch classiﬁcation results and demonstrate simultaneous material recognition and segmentation of an image by
performing dense classiﬁcation over the image with a fully
connected conditional random ﬁeld (CRF) model [ 12]. By


performing dense classiﬁcation over the image with a fully
connected conditional random ﬁeld (CRF) model [ 12]. By
replacing the fully connected layers of the CNN with convolutional layers [ 24], the computational burden is signiﬁcantly
lower than a naive sliding window approach.
In summary, we make two new contributions:
We introduce a new material dataset, MINC , and 3stage crowdsourcing pipeline for efﬁciently collecting
millions of click labels (Section 3.2).
Our new semantic segmentation method combines a
fully-connected CRF with unary predictions based on
CNN learned features (Section 4.2) for simultaneous
material recognition and segmentation.
2. Prior Work
Material Databases. Much of the early work on material
recognition focused on classifying speciﬁc instances of textures or material samples. For instance, the CUReT [ 4]
database contains 61 material samples, each captured under
205 different lighting and viewing conditions. This led to
research on the task of instance-level texture or material classiﬁcation [ 15,30], and an appreciation of the challenges of
building features that are invariant to pose and illumination.


building features that are invariant to pose and illumination.
Later, databases with more diverse examples from each ma-terial category began to appear, such as KTH-TIPS [ 9,2],
and led explorations of how to generalize from one example
of a material to another—from one sample of wood to a completely different sample, for instance. Real-world texture
attributes have also recently been explored [3].
In the domain of categorical material databases, Sharan et
al. released FMD [ 26] (described above). Subsequently,
Bell et al. released OpenSurfaces [ 1] which contains over
20,000 real-world scenes labeled with both materials and objects, using a multi-stage crowdsourcing pipeline. Because
OpenSurfaces images are drawn from consumer photos on
Flickr, material samples have real-world context, in contrast
to prior databases (CUReT, KTH-TIPS, FMD) which feature cropped stand-alone samples. While OpenSurfaces is a
good starting point for a material database, we substantially
expand it with millions of new labels.
Material recognition. Much prior work on material recognition has focused on the classiﬁcation problem (categorizing
an image patch into a set of material categories), often using


Material recognition. Much prior work on material recognition has focused on the classiﬁcation problem (categorizing
an image patch into a set of material categories), often using
hand-designed image features. For FMD, Liu et al. [17] introduced reﬂectance-based edge features in conjunction with
other general image features. Hu et al. [10] proposed features based on variances of oriented gradients. Qi et al. [20]
introduced a pairwise local binary pattern (LBP) feature.
Liet al. [16] synthesized a dataset based on KTH-TIPS2
and built a classiﬁer from LBP and dense SIFT. Timofte et
al. [29] proposed a classiﬁcation framework with minimal
parameter optimization. Schwartz and Nishino [ 23] introduced material traits that incorporate learned convolutional
auto-encoder features. Recently, Cimpoi et al. [3] developed a CNN and improved Fisher vector (IFV) classiﬁer that
achieves state-of-the-art results on FMD and KTH-TIPS2.
Finally, it has been shown that jointly predicting objects and
materials can improve performance [10, 33].
Convolutional neural networks. While CNNs have been


Finally, it has been shown that jointly predicting objects and
materials can improve performance [10, 33].
Convolutional neural networks. While CNNs have been
around for a few decades, with early successes such as
LeNet [ 14], they have only recently led to state-of-theart results in object classiﬁcation and detection, leading
to enormous progress. Driven by the ILSVRC challenge [ 21], we have seen many successful CNN architectures [ 32,24,28,27], led by the work of Krizhevsky et al.
on their SuperVision (a.k.a. AlexNet ) network [ 13], with
more recent architectures including GoogLeNet [ 28]. In addition to image classiﬁcation, CNNs are the state-of-the-art
for detection and localization of objects, with recent work
including R-CNNs [ 7], Overfeat [ 24], and VGG [ 27]. Finally, relevant to our goal of per-pixel material segmentation,
Farabet et al. [6] use a multi-scale CNN to predict the class
at every pixel in a segmentation. Oquab et al. [18] employ a
sliding window approach to localize patch classiﬁcation of
objects. We build on this body of work in deep learning to


sliding window approach to localize patch classiﬁcation of
objects. We build on this body of work in deep learning to
solve our problem of material recognition and segmentation.
2
Brick Carpet Ceramic Fabric Foliage Food Glass Hair
Leather Metal Mirror Other Painted Paper Plastic Pol. stone
Skin Sky Stone Tile Wallpaper Water Wood
Figure 2. Example patches from all 23 categories of the Materials in Context Database (MINC ). Note that we sample patches so that the
patch center is the material in question (and not necessarily the entire patch). See Table 1 for the size of each category.
3. The Materials in Context Database (MINC)
We now describe the methodology that went into building
our new material database. Why a new database? We needed
a dataset with the following properties:
Size: It should be sufﬁciently large that learning methods can generalize beyond the training set.
Well-sampled : Rare categories should be represented
with a large number of examples.
Diversity : Images should span a wide range of appearances of each material in real-world settings.
Number of categories : It should contain many different materials found in the real world.
3.1. Sources of data
We decided to start with the public, crowdsourced
OpenSurfaces dataset [ 1] as the seed for MINC since it is


3.1. Sources of data
We decided to start with the public, crowdsourced
OpenSurfaces dataset [ 1] as the seed for MINC since it is
drawn from Flickr imagery of everyday, real-world scenes
with reasonable diversity. Furthermore, it has a large number
of categories and the most samples of all prior databases.
While OpenSurfaces data is a good start, it has a few limitations. Many categories in OpenSurfaces are not well sampled. While the largest category, wood , has nearly 20K samples, smaller categories, such as water , have only tens of examples. This imbalance is due to the way the OpenSurfaces
dataset was annotated; workers on Amazon Mechanical Turk
(AMT) were free to choose any material subregion to segment. Workers often gravitated towards certain common
types of materials or salient objects, rather than being encouraged to label a diverse set of materials. Further, the
images come from a single source (Flickr).
We decided to augment OpenSurfaces with substantially
more data, especially for underrepresented material cate-gories, with the initial goal of gathering at least 10K samples
per material category. We decided to gather this data from
another source of imagery, professional photos on the interior design website Houzz (houzz.com). Our motivation for
using this different source of data was that, despite Houzz


another source of imagery, professional photos on the interior design website Houzz (houzz.com). Our motivation for
using this different source of data was that, despite Houzz
photos being more “staged” (relative to Flickr photos), they
actually represent a larger variety of materials. For instance,
Houzz photos contain a wide range of types of polished
stone. With these sources of image data, we now describe
how we gather material annotations.
3.2. Segments, Clicks, and Patches
What speciﬁc kinds of material annotations make for a
good database? How should we collect these annotations?
The type of annotations to collect is guided in large part by
the tasks we wish to generate training data for. For some
tasks such as scene recognition, whole-image labels can
sufﬁce [ 31,34]. For object detection, labeled bounding
boxes as in PASCAL are often used [ 5]. For segmentation or
scene parsing tasks, per-pixel segmentations are required [ 22,
8]. Each style of annotation comes with a cost proportional
to its complexity. For materials, we decided to focus on two
problems, guided by prior work:
Patch material classiﬁcation. Given an image patch,
what kind of material is it at the center?


problems, guided by prior work:
Patch material classiﬁcation. Given an image patch,
what kind of material is it at the center?
Full scene material classiﬁcation. Given a full image, produce a full per-pixel segmentation and labeling.
Also known as semantic segmentation orscene parsing
(but in our case, focused on materials). Note that classiﬁcation can be a component of segmentation, e.g., with
sliding window approaches.
3
(a) Which images
contain wood ?(b) Click on 3
points of wood(c) What is this
material?Figure 3. AMT pipeline schematic for collecting clicks. (a)
Workers ﬁlter by images that contain a certain material, (b)workers click on materials, and (c)workers validate click locations by
re-labeling each point. Example responses are shown in orange.
Segments. OpenSurfaces contains material segmentations—
carefully drawn polygons that enclose same-material regions.
To form the basis of MINC, we selected OpenSurfaces segments with high conﬁdence (inter-worker agreement) and
manually curated segments with low conﬁdence, giving a
total of 72K shapes. To better balance the categories, we


manually curated segments with low conﬁdence, giving a
total of 72K shapes. To better balance the categories, we
manually segmented a few hundred extra samples for sky,
foliage andwater .
Since some of the OpenSurfaces categories are difﬁcult
for humans, we consolidated these categories. We found
that many AMT workers could not disambiguate stone from
concrete ,clear plastic from opaque plastic , and granite
from marble . Therefore, we merged these into stone ,plastic ,
andpolished stone respectively. Without this merging, many
ground truth examples in these categories would be incorrect.
The ﬁnal list of 23 categories is shown in Table 1. The
category other is different in that it was created by combining
various smaller categories.
Clicks. Since we want to expand our dataset to millions
of samples, we decided to augment OpenSurfaces segments
by collecting clicks : single points in an image along with a
material label, which are much cheaper and faster to collect.
Figure 3 shows our pipeline for collecting clicks.
Initially, we tried asking workers to click on examples
of a given material in a photo. However, we found that
workers would get frustrated if the material was absent in
too many of the photos. Thus, we added an initial ﬁrst


workers would get frustrated if the material was absent in
too many of the photos. Thus, we added an initial ﬁrst
stage where workers ﬁlter out such photos. To increase
the accuracy of our labels, we verify the click labels by
asking different workers to specify the material for each
click without providing them with the label from the previous
stage.
To ensure that we obtain high quality annotations and
avoid collecting labels from workers who are not making an
effort, we include secret known answers (sentinels) in the
ﬁrst and third stages, and block workers with an accuracy
below 50% and 85% respectively. We do not use sentinels
in the second stage since it would require per-pixel ground
truth labels, and it turned out not to be necessary. Workers
generally performed all three tasks so we could identify bad
workers in the ﬁrst or third task.Patches Category Patches Category Patches Category
564,891 Wood 114,085 Polished stone 35,246 Skin
465,076 Painted 98,891 Carpet 29,616 Stone
397,982 Fabric 83,644 Leather 28,108 Ceramic
216,368 Glass 75,084 Mirror 26,103 Hair
188,491 Metal 64,454 Brick 25,498 Food
147,346 Tile 55,364 Water 23,779 Paper


216,368 Glass 75,084 Mirror 26,103 Hair
188,491 Metal 64,454 Brick 25,498 Food
147,346 Tile 55,364 Water 23,779 Paper
142,150 Sky 39,612 Other 14,954 Wallpaper
120,957 Foliage 38,975 Plastic
Table 1. MINC patch counts by category. Patches were created
from both OpenSurfaces segments and our newly collected clicks .
See Section 3.2 for details.
Material clicks were collected for both OpenSurfaces
images and the new Houzz images. This allowed us to use
labels from OpenSurfaces to generate the sentinel data; we
included 4 sentinels per task. With this streamlined pipeline
we collected 2,341,473 annotations at an average cost of
$0.00306 per annotation (stage 1: $0.02 / 40 images, stage
2: $0.10 / 50 images, 2, stage 3: $0.10 / 50 points).
Patches. Labeled segments and clicks form the core of
MINC . For training CNNs and other types of classiﬁers,
it is useful to have data in the form of ﬁxed-sized patches.
We convert both forms of data into a uniﬁed dataset format:
square image patches. We use a patch center andpatch scale


We convert both forms of data into a uniﬁed dataset format:
square image patches. We use a patch center andpatch scale
(a multiplier of the smaller image dimension) to deﬁne the
image subregion that makes a patch. For our patch classiﬁcation experiments, we use 23.3% of the smaller image
dimension. Increasing the patch scale provides more context
but reduces the spatial resolution. Later in Section 5 we
justify our choice with experiments that vary the patch scale
for AlexNet.
We place a patch centered around each click label. For
each segment , if we were to place a patch at every interior
pixel then we would have a very large and redundant dataset.
Therefore, we Poisson-disk subsample each segment , separating patch centers by at least 9.1% of the smaller image
dimension. These segments generated 655,201 patches (an
average of 9.05 patches per segment ). In total, we generated 2,996,674 labeled patches from 436,749 images. Patch
counts are shown in Table 1, and example patches from
various categories are illustrated in Figure 2.
4. Material recognition in real-world images
Our goal is to train a system that recognizes the material
at every pixel in an image. We split our training procedure
into multiple stages and analyze the performance of the


Our goal is to train a system that recognizes the material
at every pixel in an image. We split our training procedure
into multiple stages and analyze the performance of the
network at each stage. First, we train a CNN that produces a
single prediction for a given input patch. Then, we convert
the CNN into a sliding window and predict materials on a
dense grid across the image. We do this at multiple scales
and average to obtain a unary term. Finally, a dense CRF
[12] combines the unary term with fully connected pairwise
reasoning to output per-pixel material predictions. The entire
system is depicted in Figure 1, and described more below.
4
Sky
Water
FoliageTile
(a) Multiscale input (b) Probability map (12 of 23 shown) (c) Predicted labelsSliding
CNNDense
CRF
Wood
Plastic Sliding
CNNSliding
CNN
Avg.
Glass
Hair
Fabric
Polished
stone
Stone
 PaperWaterSky
Foliage
TilePaperWood
StoneWood
TileFigure 4. Pipeline for full scene material classiﬁcation. An image (a)is resized to multiple scales [1=p
2;1;p
2]. The same sliding CNN


2;1;p
2]. The same sliding CNN
predicts a probability map (b)across the image for each scale; the results are upsampled and averaged. A fully connected CRF predicts a
ﬁnal label for each pixel (c). This example shows predictions from a single GoogLeNet converted into a sliding CNN (no average pooling).
4.1. Training procedure
MINC contains 3 million patches that we split into training, validation and test sets. Randomly splitting would result
in nearly identical patches (e.g., from the same OpenSurfaces segment) being put in training and test, thus inﬂating
the test score. To prevent correlation, we group photos into
clusters of near-duplicates, then assign each cluster to one
of train, validate or test. We make sure that there are at least
75segments of each category in the test set to ensure there
are enough segments to evaluate segmentation accuracy. To
detect near-duplicates, we compare AlexNet CNN features
computed from each photo (see the supplemental for details).
For exact duplicates, we discard all but one of the copies.
We train all of our CNNs by ﬁne-tuning the network
starting from the weights obtained by training on 1.2 million images from ImageNet (ILSVRC2012). When training


starting from the weights obtained by training on 1.2 million images from ImageNet (ILSVRC2012). When training
AlexNet , we use stochastic gradient descent with batchsize
128, dropout rate 0.5, momentum 0.9, and a base learning
rate of 103that decreases by a factor of 0.25 every 50,000
iterations. For GoogLeNet , we use batchsize 69, dropout 0.4,
and learning rate t= 104p
1t=250000 for iteration t.
Our training set has a different number of examples per
class, so we cycle through the classes and randomly sample
an example from each class. Failing to properly balance the
examples results in a 5.7% drop in mean class accuracy (on
the validation set). Further, since it has been shown to reduce
overﬁtting, we randomly augment samples by taking crops
(227227out of 256256), horizontal mirror ﬂips, spatial
scales in the range [1=p
2;p
2], aspect ratios from 3:4 to 4:3,
and amplitude shifts in [0:95;1:05]. Since we are looking at
local regions, we subtract a per-channel mean (R: 124, G:


and amplitude shifts in [0:95;1:05]. Since we are looking at
local regions, we subtract a per-channel mean (R: 124, G:
117, B: 104) rather than a mean image [13].
4.2. Full scene material classiﬁcation
Figure 4 shows an overview of our method for simultaneously segmenting and recognizing materials. Given a
CNN that can classify individual points in the image, we
convert it to a sliding window detector and densely classify
a grid across the image. Speciﬁcally, we replace the last
fully connected layers with convolutional layers, so that the
network is fully convolutional and can classify images ofany shape. After conversion, the weights are ﬁxed and not
ﬁne-tuned. With our converted network, the strides of each
layer cause the network to output a prediction every 32 pixels. We obtain predictions every 16 pixels by shifting the
input image by half-strides (16 pixels). While this appears to
require 4x the computation, Sermanet et al. [24] showed that
the convolutions can be reused and only the pool5 through
fc8 layers need to be recomputed for the half-stride shifts.
Adding half-strides resulted in a minor 0.2% improvement
in mean class accuracy across segments (after applying the


Adding half-strides resulted in a minor 0.2% improvement
in mean class accuracy across segments (after applying the
dense CRF, described below), and about the same mean class
accuracy at click locations.
The input image is resized so that a patch maps to a
256x256 square. Thus, for a network trained at patch scale
s, the resized input has smaller dimension d= 256=s. Note
thatdis inversely proportional to scale, so increased context
leads to lower spatial resolution. We then add padding so
that the output probability map is aligned with the input
when upsampled. We repeat this at 3 different scales (smaller
dimensiond=p
2,d,dp
2), upsample each output probability
map with bilinear interpolation, and average the predictions.
To make the next step more efﬁcient, we upsample the output
to a ﬁxed smaller dimension of 550.
We then use the dense CRF of Kr¨ahenb ¨uhlet al. [12] to
predict a label at every pixel, using the following energy:
E(xjI) =X
i i(xi) +X
i<j ij(xi;xj) (1)
 i(xi) =logpi(xi) (2)


i i(xi) +X
i<j ij(xi;xj) (1)
 i(xi) =logpi(xi) (2)
 ij(xi;xj) =wp(xi6=xj)k(fifj) (3)
where iis the unary energy (negative log of the aggregated softmax probabilities) and  ijis the pairwise term
that connects every pair of pixels in the image. We use a
single pairwise term with a Potts label compatibility term
weighted by wpand unit Gaussian kernel k. For the featuresfi, we convert the RGB image to L*a*b*and use color
(IL
i;Ia
i;Ib
i)and position (px;py)as pairwise features for
each pixel: fi=h
px
i
pd;py
i
pd;IL
i
L;Ia
i
ab;Ib
i
abi
, wheredis the
smaller image dimension. Figure 4 shows an example unary
termpiand the resulting segmentation x.
5
Figure 5. Varying patch scale . We train/test patches of different
scales (the patch locations do not vary). The optimum is a trade-off


5
Figure 5. Varying patch scale . We train/test patches of different
scales (the patch locations do not vary). The optimum is a trade-off
between context and spatial resolution. CNN: AlexNet.
Architecture Validation Test
AlexNet [13] 82.2% 81.4%
GoogLeNet [28] 85.9% 85.2%
VGG-16 [27] 85.6% 84.8%
Table 2. Patch material classiﬁcation results. Mean class accuracy for different CNNs trained on MINC. See Section 5.1.
Sky 97.3% Food 90.4% Wallpaper 83.4% Glass 78.5%
Hair 95.3% Leather 88.2% Tile 82.7% Fabric 77.8%
Foliage 95.1% Other 87.9% Ceramic 82.7% Metal 77.7%
Skin 93.9% Pol. stone 85.8% Stone 82.7% Mirror 72.0%
Water 93.6% Brick 85.1% Paper 81.8% Plastic 70.9%
Carpet 91.6% Painted 84.2% Wood 81.3%
Table 3. Patch test accuracy by category. CNN: GoogLeNet . See
the supplemental material for a full confusion matrix.
5. Experiments and Results


Table 3. Patch test accuracy by category. CNN: GoogLeNet . See
the supplemental material for a full confusion matrix.
5. Experiments and Results
5.1. Patch material classiﬁcation
In this section, we evaluate the effect of many different
design decisions for training methods for material classiﬁcation and segmentation, including various CNN architectures,
patch sizes, and amounts of data.
CNN Architectures. Our ultimate goal is full material segmentation, but we are also interested in exploring which
CNN architectures give the best results for classifying single patches. Among the networks and parameter variations we tried we found the best performing networks were
AlexNet [13],VGG-16 [27] and GoogLeNet [28].AlexNet
andGoogLeNet are re-implementations by BVLC [ 11], and
VGG-16 is conﬁguration D (a 16 layer network) of [ 27].
All models were obtained from the Caffe Model Zoo [ 11].
Our experiments use AlexNet for evaluating material classiﬁcation design decisions and combinations of AlexNet and
GoogLeNet for evaluating material segmentation. Tables 2
and 3 summarize patch material classiﬁcation results on our
dataset. Figure 10 shows correct and incorrect predictions
made with high conﬁdence.


and 3 summarize patch material classiﬁcation results on our
dataset. Figure 10 shows correct and incorrect predictions
made with high conﬁdence.
Input patch scale. To classify a point in an image we must
decide how much context to include around it. The context,
expressed as a fraction of image size, is the patch scale. A
priori, it is not clear which scale is best since small patches
have better spatial resolution, but large patches have more
0.5e6 1e6 1.5e6 2e6 2.5e6
Number of training patches0.600.650.700.750.800.85Mean class accuracy (test set)66.1%74.8%77.9%79.3%80.2%80.9% 81.2% 81.4%Equal size
79.6%Figure 6. Varying database size. Patch accuracy when trained on
random subsets of MINC .Equal size is using equal samples per
category (size determined by smallest category). CNN: AlexNet.
Peak accuracy
per category
Figure 7. Accuracy vs patch scale by category . Dots: peak accuracy for each category; colored lines: sky,wallpaper ,mirror ; gray
lines: other categories. CNN: AlexNet . While most materials are


lines: other categories. CNN: AlexNet . While most materials are
optimally recognized at 23.3% or 32% patch scale, recognition of
sky,wallpaper andmirror improve with increasing context.
contextual information. Holding patch centers ﬁxed we varied scale and evaluated classiﬁcation accuracy with AlexNet .
Results and a visualization of patch scales are shown in Figure 5. Scale 32% performs the best. Individual categories
had peaks at middle scale with some exceptions; we ﬁnd
thatmirror ,wallpaper andskyimprove with increasing context (Figure 7). We used 23.3% (which has nearly the same
accuracy but higher spatial resolution) for our experiments.
Dataset size. To measure the effect of size on patch classiﬁcation accuracy we trained AlexNet with patches from
randomly sampled subsets of all 369,104 training images
and tested on our full test set (Figure 6). As expected, using
more data improved performance. In addition, we still have
not saturated performance with 2.5 million training patches;
even higher accuracies may be possible with more training
data (though with diminishing returns).
Dataset balance. Although we’ve shown that more data is
better we also ﬁnd that a balanced dataset is more effective.
We trained AlexNet with all patches of our smallest category


better we also ﬁnd that a balanced dataset is more effective.
We trained AlexNet with all patches of our smallest category
(wallpaper ) and randomly sampled the larger categories
(the largest, wood , being 40x larger) to be equal size. We
then measured mean class accuracy on the same full test
set. As shown in Figure 6, “Equal size” is more accurate
than a dataset of the same size and just 1.7% lower than
the full training set (which is 9x larger). This result further
demonstrates the value of building up datasets in a balanced
manner, focusing on expanding the smallest, least common
categories.
6
Brick
WoodCarpet
WaterWallpaperTileStoneSkySkinPolished stonePlasticPaperPainted
OtherMirrorMetalLeatherHairGlassFoodFoliageFabricCeramicFigure 8. Full-scene material classiﬁcation examples: high-accuracy test set predictions by our method. CNN: GoogLeNet (with the
average pooling layer removed). Right: legend for material colors. See Table 4 for quantitative evaluation.
Input image (a)Labels from CRF (b)Labels from CRF
(test set) trained on segments trained on clicks


Input image (a)Labels from CRF (b)Labels from CRF
(test set) trained on segments trained on clicks
Figure 9. Optimizing for click accuracy leads to sloppy boundaries. In(a), we optimize for mean class accuracy across segments,
resulting in high quality boundaries. In (b), we optimize for mean
class accuracy at click locations. Since the clicks are not necessarily close to object boundaries, there is no penalty for sloppy
boundaries. CNN: GoogLeNet (without average pooling).
5.2. Full scene material segmentation
The full test set for our patch dataset contains 41,801
photos, but most of them contain only a few labels. Since
we want to evaluate the per-pixel classiﬁcation performance,
we select a subset of 5,000 test photos such that each photo
contains a large number of segments and clicks, and small
categories are well sampled. We greedily solve for the best
such set of photos. We similarly select 2,500 of 25,844
validation photos. Our splits for all experiments are included
online with the dataset. To train the CRF for our model, we
try various parameter settings ( p,ab,L,wp) and select the
model that performs best on the validation set. In total, we


try various parameter settings ( p,ab,L,wp) and select the
model that performs best on the validation set. In total, we
evaluate 1799 combinations of CNNs and CRF parameters.
See the supplemental material for a detailed breakdown.
We evaluate multiple versions of GoogLeNet : both the
original architecture and a version with the average pooling
layer (at the end) changed to 5x5, 3x3, and 1x1 (i.e. no
average pooling). We evaluate AlexNet trained at multiple
patch scales (Figure 5). When using an AlexNet trained
at a different scale, we keep the same scale for testing. We
also experiment with ensembles of GoogLeNet andAlexNet ,Architecture(a)Segments only (b)Clicks only
Class Total Class Total
AlexNet Scale: 11.6% 64.3% 72.6% 79.9% 77.2%
AlexNet Scale: 23.3% 69.6% 76.6% 83.3% 81.1%
AlexNet Scale: 32.0% 70.1% 77.1% 83.2% 80.7%
AlexNet Scale: 46.5% 69.6% 75.4% 80.8% 77.7%


AlexNet Scale: 46.5% 69.6% 75.4% 80.8% 77.7%
AlexNet Scale: 66.2% 67.7% 72.0% 77.2% 72.6%
GoogLeNet 7x7 avg. pool 64.4% 71.6% 63.6% 63.4%
GoogLeNet 5x5 avg. pool 67.6% 74.6% 70.9% 69.8%
GoogLeNet 3x3 avg. pool 70.4% 77.7% 76.1% 74.7%
GoogLeNet No avg. pool 70.4% 78.8% 79.1% 77.4%
Ensemble 2 CNNs 73.1% 79.8% 84.5% 83.1%
Ensemble 3 CNNs 73.1% 79.3% 85.9% 83.5%
Ensemble 4 CNNs 72.1% 78.4% 85.8% 83.2%
Ensemble 5 CNNs 71.7% 78.3% 85.5% 83.2%
Table 4. Full scene material classiﬁcation results. Mean class
and total accuracy on the test set. When training, we optimize the
CRF parameters for mean class accuracy, but report both mean class


and total accuracy on the test set. When training, we optimize the
CRF parameters for mean class accuracy, but report both mean class
and total accuracy (mean accuracy across all examples). In one
experiment (a), we train and test only on segments; in a separate
experiment (b), we train and test only on clicks. Accuracies for
segments are averaged across all pixels that fall in that segment.
combined with either arithmetic or geometric mean.
Since we have two types of data, clicks andsegments , we
run two sets of experiments: (a) we train and test only on
segments, and in a separate experiment (b) we train and test
only on clicks. These two training objectives result in very
different behavior, as illustrated in Figure 9. In experiment
(a), the accuracy across segments are optimized, producing
clean boundaries. In experiment (b), the CRF maximizes
accuracy only at click locations, thus resulting in sloppy
boundaries. As shown in Table 4, the numerical scores for
the two experiments are also very different: segments are
more challenging than clicks. While clicks are sufﬁcient to
train a CNN, they are not sufﬁcient to train a CRF.
Focusing on segmentation accuracy, we see from Table 4(a) that our best single model is GoogLeNet without


Focusing on segmentation accuracy, we see from Table 4(a) that our best single model is GoogLeNet without
average pooling (6% better than with pooling). The best
ensemble is 2 CNNs: GoogLeNet (no average pooling) and
AlexNet (patch scale: 46.5%), combined with arithmetic
mean. Larger ensembles perform worse since we are aver7
Correct
Fabric (99%) Foliage (99%) Food (99%) Leather (99%)Correct
Metal (99%) Mirror (99%) Painted (99%) Plastic (99%)Incorrect
T: Wood T: Polished stone T: Water T: Fabric
P: Stone (90%) P: Water (35%) P: Carpet (27%) P: Foliage (67%)
Figure 10. High conﬁdence predictions. Top two rows: correct
predictions. Bottom row: incorrect predictions ( T: true, P: predicted). Percentages indicate conﬁdence (the predictions shown are
at least this conﬁdent). CNN: GoogLeNet.
aging worse CNNs. In Figure 8, we show example labeling
results on test images.
5.3. Comparing MINC to FMD
Compared to FMD, the size and diversity of MINC is


results on test images.
5.3. Comparing MINC to FMD
Compared to FMD, the size and diversity of MINC is
valuable for classifying real-world imagery. Table 5 shows
the effect of training on all of FMD and testing on MINC
(and vice versa). The results suggests that training on FMD
alone is not sufﬁcient for real-world classiﬁcation. Though it
may seem that our dataset is “easy,” since the best classiﬁcations scores are lower for FMD than for MINC , we ﬁnd that
difﬁculty is in fact closely tied to dataset size (Section 5.1).
Taking 100 random samples per category, AlexNet achieves
54.20.7% on MINC (64.6 1.3% when considering only
the 10 FMD categories) and 66.5% on FMD.
5.4. Comparing CNNs with prior methods
Cimpoi [ 3] is the best prior material classiﬁcation method
on FMD. We ﬁnd that by replacing DeCAF with oversampled AlexNet features we can improve on their FMD results.
We then show that on MINC , a ﬁnetuned CNN is even better.


We then show that on MINC , a ﬁnetuned CNN is even better.
To improve on [ 3], we take their SIFT IFV, combine it
with AlexNet fc7 features, and add oversampling [ 13] (see
supplemental for details). With a linear SVM we achieve
69.60.3% on FMD. Previous results are listed in Table 6.
Having found that SIFT IFV+fc7 is the new best on
FMD, we compare it to a ﬁnetuned CNN on a subset of MINC (2500 patches per category, one patch per
photo). Fine-tuning AlexNet achieves 76.00.2% whereasTest
FMD MINC
TrainFMD 66.5% 26.1%
MINC 41.7% 85.0%(10 categories
in common)
Table 5. Cross-dataset experiments. We train on one dataset and
test on another dataset. Since MINC contains 23 categories, we
limit MINC to the 10 categories in common. CNN: AlexNet.
Method Accuracy Trials
Sharan et al. [25] 57.10.6% 14 splits
Cimpoi et al. [3] 67.10.4% 14 splits
Fine-tuned AlexNet 66.51.5% 5 folds


Cimpoi et al. [3] 67.10.4% 14 splits
Fine-tuned AlexNet 66.51.5% 5 folds
SIFT IFV+fc7 69.60.3% 10 splits
Table 6. FMD experiments. By replacing DeCAF features with
oversampled AlexNet features we improve on the best FMD result.
SIFT IFV+fc7 achieves 67.4 0.5% with a linear SVM
(oversampling, 5 splits). This experiment shows that
a ﬁnetuned CNN is a better method for MINC than
SIFT IFV+fc7.
6. Conclusion
Material recognition is a long-standing, challenging problem. We introduce a new large, open, material database,
MINC , that includes a diverse range of materials of everyday scenes and staged designed interiors, and is at least an
order of magnitude larger than prior databases. Using this
large database we conduct an evaluation of recent deep learning algorithms for simultaneous material classiﬁcation and
segmentation, and achieve results that surpass prior attempts
at material recognition.
Some lessons we have learned are:
Training on a dataset which includes the surrounding
context is crucial for real-world material classiﬁcation.


at material recognition.
Some lessons we have learned are:
Training on a dataset which includes the surrounding
context is crucial for real-world material classiﬁcation.
Labeled clicks are cheap and sufﬁcient to train a CNN
alone. However, to obtain high quality segmentation
results, training a CRF on polygons results in much
better boundaries than training on clicks.
Many future avenues of work remain. Expanding the
dataset to a broader range of categories will require new
ways to mine images that have more variety, and new annotation tasks that are cost-effective. Inspired by attributes
for textures [ 3], in the future we would like to identify material attributes and expand our database to include them. We
also believe that further exploration of joint material and object classiﬁcation and segmentation will be fruitful [ 10] and
lead to improvements in both tasks. Our database, trained
models, and all experimental results are available online at
http://minc.cs.cornell.edu/.
Acknowledgements. This work was supported in part by
Google, Amazon AWS for Education, a NSERC PGS-D
scholarship, the National Science Foundation (grants IIS1149393, IIS-1011919, IIS-1161645), and the Intel Science
and Technology Center for Visual Computing.
8
References


and Technology Center for Visual Computing.
8
References
[1]S. Bell, P. Upchurch, N. Snavely, and K. Bala. OpenSurfaces:
A richly annotated catalog of surface appearance. ACM Trans.
on Graphics (SIGGRAPH) , 32(4), 2013. 1, 2, 3
[2]B. Caputo, E. Hayman, and P. Mallikarjuna. Class-speciﬁc
material categorisation. In ICCV , pages 1597–1604, 2005. 2
[3]M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and
A. Vedaldi. Describing textures in the wild. In CVPR , pages
3606–3613. IEEE, 2014. 2, 8
[4]K. J. Dana, B. Van Ginneken, S. K. Nayar, and J. J. Koenderink. Reﬂectance and texture of real-world surfaces. ACM
Transactions on Graphics (TOG) , 18(1):1–34, 1999. 2
[5]M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The Pascal Visual Object Classes (VOC)


and A. Zisserman. The Pascal Visual Object Classes (VOC)
Challenge. IJCV , 88(2):303–338, June 2010. 3
[6]C. Farabet, C. Couprie, L. Najman, and Y . LeCun. Learning
hierarchical features for scene labeling. PAMI , 35(8):1915–
1929, 2013. 2
[7]R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich
feature hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014. 2
[8]S. Gould, R. Fulton, and D. Koller. Decomposing a scene
into geometric and semantically consistent regions. In ICCV ,
2009. 3
[9]E. Hayman, B. Caputo, M. Fritz, and J. olof Eklundh. On the
signiﬁcance of real-world conditions for material classiﬁcation. In ECCV , 2004. 2
[10] D. Hu, L. Bo, and X. Ren. Toward robust material recognition
for everyday objects. In BMVC , pages 1–11. Citeseer, 2011.
1, 2, 8


for everyday objects. In BMVC , pages 1–11. Citeseer, 2011.
1, 2, 8
[11] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional
architecture for fast feature embedding. In Proceedings of
the ACM International Conference on Multimedia , pages 675–
678. ACM, 2014. 6
[12] P. Kr ¨ahenb ¨uhl and V . Koltun. Parameter learning and convergent inference for dense random ﬁelds. In ICML , pages
513–521, 2013. 2, 4, 5
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
Advances in neural information processing systems , pages
1097–1105, 2012. 2, 5, 6, 8
[14] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel. Backpropagation


Howard, W. Hubbard, and L. D. Jackel. Backpropagation
applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989. 2
[15] T. Leung and J. Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons.
IJCV , 43(1):29–44, June 2001. 2
[16] W. Li and M. Fritz. Recognizing materials from virtual examples. In ECCV , pages 345–358. Springer, 2012. 2
[17] C. Liu, L. Sharan, E. H. Adelson, and R. Rosenholtz. Exploring features in a bayesian framework for material recognition.
InCVPR , pages 239–246. IEEE, 2010. 1, 2
[18] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and
transferring mid-level image representations using convolutional neural networks. In CVPR , 2014. 2[19] G. Patterson, C. Xu, H. Su, and J. Hays. The SUN Attribute
Database: Beyond Categories for Deeper Scene Understanding. IJCV , 108(1-2):59–81, 2014. 1


Database: Beyond Categories for Deeper Scene Understanding. IJCV , 108(1-2):59–81, 2014. 1
[20] X. Qi, R. Xiao, J. Guo, and L. Zhang. Pairwise rotation
invariant co-occurrence local binary pattern. In ECCV , pages
158–171. Springer, 2012. 1, 2
[21] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg,
and L. Fei-Fei. ImageNet Large Scale Visual Recognition
Challenge, 2014. 1, 2
[22] B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman. LabelMe: A database and web-based tool for image
annotation. IJCV , 77(1-3):157–173, May 2008. 3
[23] G. Schwartz and K. Nishino. Visual material traits: Recognizing per-pixel material context. In Proceedings of the
International Conference on Computer Vision Workshops (ICCVW) , pages 883–890. IEEE, 2013. 2


International Conference on Computer Vision Workshops (ICCVW) , pages 883–890. IEEE, 2013. 2
[24] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization
and detection using convolutional networks. In International
Conference on Learning Representations (ICLR2014) . CBLS,
April 2014. 2, 5
[25] L. Sharan, C. Liu, R. Rosenholtz, and E. Adelson. Recognizing materials using perceptually inspired features. IJCV ,
2013. 1, 8
[26] L. Sharan, R. Rosenholtz, and E. Adelson. Material perception: What can you see in a brief glance? Journal of Vision ,
9(8):784–784, 2009. 1, 2
[27] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556 , 2014. 2, 6
[28] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov,


[28] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov,
D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper
with convolutions. CVPR , 2015. 2, 6
[29] R. Timofte and L. J. Van Gool. A training-free classiﬁcation
framework for textures, writers, and materials. In BMVC ,
pages 1–12, 2012. 2
[30] M. Varma and A. Zisserman. A statistical approach to texture
classiﬁcation from single images. IJCV , 62(1-2):61–81, Apr.
2005. 2
[31] J. Xiao, K. A. Ehinger, J. Hays, A. Torralba, and A. Oliva.
SUN Database: Exploring a large collection of scene categories. IJCV , 2014. 1, 3
[32] M. D. Zeiler and R. Fergus. Visualizing and understanding
convolutional networks. In ECCV , pages 818–833. Springer,
2014. 2
[33] S. Zheng, M.-M. Cheng, J. Warrell, P. Sturgess, V . Vineet,


2014. 2
[33] S. Zheng, M.-M. Cheng, J. Warrell, P. Sturgess, V . Vineet,
C. Rother, and P. H. Torr. Dense semantic image segmentation
with objects and attributes. In CVPR , pages 3214–3221. IEEE,
2014. 2
[34] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva.
Learning deep features for scene recognition using Places
database. NIPS , 2014. 1, 3
9


DeepEdge: A Multi-Scale Bifurcated Deep Network
for Top-Down Contour Detection
Gedas Bertasius
University of Pennsylvania
gberta@seas.upenn.eduJianbo Shi
University of Pennsylvania
jshi@seas.upenn.eduLorenzo Torresani
Dartmouth College
lt@dartmouth.edu
Abstract
Contour detection has been a fundamental component
in many image segmentation and object detection systems.
Most previous work utilizes low-level features such as texture or saliency to detect contours and then use them as cues
for a higher-level task such as object detection. However,
we claim that recognizing objects and predicting contours
are two mutually related tasks. Contrary to traditional approaches, we show that we can invert the commonly established pipeline: instead of detecting contours with low-level
cues for a higher-level recognition task, we exploit objectrelated features as high-level cues for contour detection.
We achieve this goal by means of a multi-scale deep network that consists of ﬁve convolutional layers and a bifurcated fully-connected sub-network. The section from the input layer to the ﬁfth convolutional layer is ﬁxed and directly

Summary:
Contour detection has been a fundamental component in many image segmentation
and object detection systems. Most previous work utilizes low-level features
such as texture or saliency to detect contours and then use them as cues for a
higher-level task such as object detection. However, we claim that recognizing
objects and predicting contours are two mutually related tasks. Contrary to
traditional approaches, we show that we can invert the commonly established
pipeline: instead of detecting contours with low-level cues for a higher-level
recognition task, we exploit object-related features as high-level cues for
contour detection.
  We achieve this goal by means of a multi-scale deep network that consists of
five convolutional layers and a bifurcated fully-connected sub-network. The
section from the input layer to the fifth convolutional layer is fixed and
directly lifted from a pre-trained network optimized over a large-scale object
classification task. This section of the network is applied to four different
scales of the image input. These four parallel and identical streams are then
attached to a bifurcated sub-network consisting of two independently-trained
branches. One branch learns to predict the contour likelihood (with a
classification objective) whereas the other branch is trained to learn the
fraction of human labelers agreeing about the contour presence at a given point
(with a regression criterion).
  We show that without any feature engineering our multi-scale deep learning
approach achieves state-of-the-art results in contour detection.

Gedas Bertasius
Jianbo Shi
Lorenzo Torresani

Accepted to CVPR 2015


lifted from a pre-trained network optimized over a largescale object classiﬁcation task. This section of the network
is applied to four different scales of the image input. These
four parallel and identical streams are then attached to
a bifurcated sub-network consisting of two independentlytrained branches. One branch learns to predict the contour likelihood (with a classiﬁcation objective) whereas the
other branch is trained to learn the fraction of human labelers agreeing about the contour presence at a given point
(with a regression criterion).
We show that without any feature engineering our multiscale deep learning approach achieves state-of-the-art results in contour detection.
1. Introduction
Contour detection is typically considered a low-level
problem, and used to aid higher-level tasks such as object
detection [1, 3, 31, 24]. However, it can be argued that
the tasks of detecting objects and predicting contours are
closely related. For instance, given the contours we can
easily infer which objects are present in the image. Con0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
RecallPrecision

[F=.80] Human

Accepted to CVPR 2015


RecallPrecision

[F=.80] Human
[F=.753] DeepEdge
[F=.753] N4−fields [10]
[F=.747] MCG [2]
[F=.746] SE [8]
[F=.739] SCG [27]
[F=.737] PMI [14]
[F=.727] Sketch Tokens [19]
[F=.726] gPb−owt−ucm [1]Figure 1: Contour detection accuracy on the BSDS500
dataset. Our method attains higher average precision compared to prior methods and state-of-the-art F-score. At low
recall, DeepEdge achieves nearly 100% precision..
versely, if we are given exact locations of the objects we
could predict contours just as easily. A commonly established pipeline in computer vision starts with with low-level
contour prediction and then moves up to higher-level object
detection. However, since we claim that these two tasks are
mutually related, we propose to invert this process. Instead
of using contours as low-level cues for object detection, we
want to use object-speciﬁc information as high-level cues
for contour detection. Thus, in a sense our scheme can be

Accepted to CVPR 2015


want to use object-speciﬁc information as high-level cues
for contour detection. Thus, in a sense our scheme can be
viewed as a top-down approach where object-level cues inform the low-level contour detection process.
In this work, we present a uniﬁed multi-scale deep learning approach that uses higher-level object information to
1arXiv:1412.1123v3  [cs.CV]  23 Apr 2015
predict contours. Speciﬁcally, we present a front-to-end
convolutional architecture where contours are learned directly from raw pixels. Our proposed deep learning architecture reuses features computed by the ﬁrst ﬁve convolutional layers of the network of Krizhevsky et al. [17]. We refer to this network as the KNet . Because the KNet has been
trained for object classiﬁcation, reusing its features enable
our method to incorporate object-level information for contour prediction. In the experimental section, we will show
that this high-level object information greatly improves contour detection results.
Furthermore, our deﬁned architecture operates on multiple scales simultaneously and combines local and global
information from the image, which leads to signiﬁcantly
improved contour detection accuracy rates.

Accepted to CVPR 2015


information from the image, which leads to signiﬁcantly
improved contour detection accuracy rates.
We connect the features computed by the convolutional
layers of the KNet at four different scales of the input with
alearned subnetwork that bifurcates into two branches (the
architecture of our model is illustrated in Fig. 2).
What should the learning objective be? When a human
observer decides if a pixel is a boundary edge, a number
of supporting evidence is used with object level reasoning.
While it is impossible to record such information, we do
have the fraction of observers in agreement for each pixel.
We argue that a learning objective that predicts the fraction
of human labelers in agreement can mimic human reasoning
better.
Thus, in the bifurcated sub-network we optimize the two
branches with different learning objectives. The weights in
one branch are optimized with an edge classiﬁcation objective, while the other branch is trained to predict the fraction of human labelers in agreement, i.e., using a regression
criterion. We show that predictions from the classiﬁcation
branch yield high edge recall, while the outputs of the regression branch have high precision. Thus, fusing these two
outputs allows us to obtain excellent results with respect to

Accepted to CVPR 2015


branch yield high edge recall, while the outputs of the regression branch have high precision. Thus, fusing these two
outputs allows us to obtain excellent results with respect to
both metrics and produce state-of-the-art F-score and average precision.
In summary, the use of higher-level object features, independent optimization of edge classiﬁcation and regression
objectives, as well as a uniﬁed multi-scale architecture are
the key characteristics that allow our method to achieve the
state-of-the-art in contour detection (see Fig. 1).
2. Related Work
Deep Learning. In the recent years, deep convolutional
networks have achieved remarkable results in a wide array
of computer vision tasks [32, 25, 33, 17]. However, thus
far, applications of convolutional networks focused on highlevel vision tasks such as face recognition, image classiﬁcation, pose estimation or scene labeling [32, 25, 33, 17]. Excellent results in these tasks beg the question whether convolutional networks could perform equally well in lowerFigure 2: Visualization of multi-scale DeepEdge network
architecture. To extract candidate contour points, we run the
Canny edge detector. Then, around each candidate point,
we extract patches at four different scales and simultaneously run them through the ﬁve convolutional layers of the

Accepted to CVPR 2015


Canny edge detector. Then, around each candidate point,
we extract patches at four different scales and simultaneously run them through the ﬁve convolutional layers of the
KNet [17]. We connect these convolutional layers to two
separately-trained network branches. The ﬁrst branch is
trained for classiﬁcation, while the second branch is trained
as a regressor. At testing time, the scalar outputs from these
two sub-networks are averaged to produce the ﬁnal score.
level vision tasks such as contour detection. In this paper,
we present a convolutional architecture that achieves stateof-the-art results in a contour detection task, thus demonstrating that convolutional networks can be applied successfully for lower-level vision tasks as well.
Edge Detection. Most of the contour detection methods
can be divided into two branches: local and global methods. Local methods perform contour detection by reasoning about small patches inside the image. Some recent
local methods include sketch tokens [18] and structured
edges [7], Both of these methods are trained in a supervised fashion using a random forest classiﬁer. Sketch tokens [18] pose contour detection as a multi-class classiﬁcation task and predicts a label for each of the pixels individually. Structured edges [7], on the other hand, attempt to

Accepted to CVPR 2015


predict the labels of multiple pixels simultaneously.
Global methods predict contours based on the information from the full image. Some of the most successful approaches in this genre are the MCG detector [2], gPb detector [1] and sparse code gradients [26]. While sparse
code gradients use supervised SVM learning [4], both gPb
and MCG rely on some form of spectral methods. Other
spectral-based methods include Normalized Cuts [30] and
PMI [13].
Recently, there have also been attempts to apply deep
learning methods to the task of contour detection. While
SCT [20] is a sparse coding approach, both N4ﬁelds [9]
and DeepNet [16] use Convolutional Neural Networks
(CNNs) to predict contours. N4ﬁelds rely on dictionary
a) Input Image
 c)2ndlayer
 d)3rdlayer
 e)4thlayer
 e)5thlayer
Figure 3: Visualization of the activation values at the selected convolutional ﬁlters of the KNet (ﬁlters are resized to the
original image dimensions). The ﬁlters in the second layer ﬁre on oriented edges inside the image. The third and fourth

Accepted to CVPR 2015


original image dimensions). The ﬁlters in the second layer ﬁre on oriented edges inside the image. The third and fourth
convolutional layers produce an outline of the shape of the object. The ﬁfth layer ﬁres on the speciﬁc parts of the object.
learning and the use of the Nearest Neighbor algorithm
within a CNN framework while DeepNet uses a traditional
CNN architecture to predict contours.
In comparison to these prior approaches, our work offers
several contributions. First, we deﬁne a novel multi-scale
bifurcated CNN architecture that enables our network to
achieve state-of-the-art contour detection results. Second,
we avoid manual feature engineering by learning contours
directly from raw data. Finally, we believe that we are the
ﬁrst to propose the use of high-level object features for contour detection, thus inverting the traditional pipeline relying on low-level cues for mid-level feature extraction. Our
experiments show that this top-down approach for contour
detection yields state-of-the-art results.
3. The DeepEdge Network
In this section, we describe our proposed deep learning approach for contour detection. For simplicity, we ﬁrst

Accepted to CVPR 2015


3. The DeepEdge Network
In this section, we describe our proposed deep learning approach for contour detection. For simplicity, we ﬁrst
present our architecture in the single-scale scenario (subsection 3.1) and then discuss how to take advantage of multiple
scales (subsection 3.2).
3.1. Single-Scale Architecture
Selection of Candidate Edge Points. To extract a set
of candidate contours with high recall we apply the Canny
edge detector [5] to the input image. For each of these
points we then extract a patch of ﬁxed size such that our
candidate point is the center of the patch. Patches that do
not ﬁt into the image boundaries are padded with the mirror
reﬂections of itself.
Extraction of High-Level Features. We then resize the
patch of ﬁxed size to match the input dimensions of the
KNet [17] and use this network to extract object-level features. The KNet is an appropriate model for our setting as it
has been trained over a large number of object classes (the
1000 categories of the ImageNet dataset [28]) and thus captures features that are generic and useful for many object
categories. While such features have been optimized for the
task of object class recognition, they have been shown to

Accepted to CVPR 2015


categories. While such features have been optimized for the
task of object class recognition, they have been shown to
be highly effective for other image-analysis tasks, includ-ing object detection [10], attribute prediction [34], and image style recognition [15]. The network was trained on 1:2
million images and it includes more than 60million parameters. Its architecture consists of 5convolutional layers and
3fully connected layers. As we intend to use the KNet as
a feature extractor for boundary detection, we utilize only
the ﬁrst 5convolutional layers, which preserve explicit location information before the “spatial scrambling” of the
fully connection layers (note that a spatially variant representation is crucially necessary to predict the presence of
contours at individual pixels). The ﬁrst two KNet convolutional layers learn low-level information. As we move
into the deeper layers, however, we observe that the network
learns higher-level object information. The second convolutional layer seems to encode coherent edge structures. The
third convolutional layer ﬁres at locations corresponding to
prototypical object shapes. The fourth layer appears to generate high responses for full shapes of the object, whereas
the ﬁfth layer ﬁres on the speciﬁc object parts (See Fig. 3).

Accepted to CVPR 2015


the ﬁfth layer ﬁres on the speciﬁc object parts (See Fig. 3).
In order to obtain a representation that captures this hierarchical information, we perform feature extraction at each
of the ﬁve convolutional layers, as shown in Fig. 5. Specifically, we consider a small sub-volume of the feature map
stack produced at each layer. The sub-volume is centered
at the center of the patch in order to assess the presence of
a contour in a small area around the candidate point. We
then perform max,average , and center pooling on this subvolume. This yields a feature descriptor of size 3Fwhere
Fis the number of feature maps computed by the convolutional layer. While max and average pooling are well established operations in deep learning, we deﬁne center pooling
as selecting the center-value from each of the feature maps.
The motivation for center pooling is that for each candidate
point we want to predict the contour presence at that particular point. Because the candidate point is located at the center of the input patch, center pooling extracts the activation
value from the location that corresponds to our candidate
point location.
A Bifurcated Sub-Network. We connect the feature
maps computed via pooling from the ﬁve convolutional

Accepted to CVPR 2015


point location.
A Bifurcated Sub-Network. We connect the feature
maps computed via pooling from the ﬁve convolutional
layers to two separately-trained network branches. Each
Figure 4: A few samples of ground truth data illustrating the difference between the classiﬁcation (ﬁrst row) and the regression
(second row) objectives. The classiﬁcation branch is trained to detect contours that are marked by at least one of the human
annotators. Conversely, the regression branch is optimized to the contour values that depict the fraction of human annotators
agreeing on the contour.
branch consists of two fully-connected layers. The ﬁrst
branch is trained using binary labels, i.e., to perform contour classiﬁcation. This branch is making less selective predictions by classifying whether a given point is a contour or
not. In a sense, this classiﬁcation branch abstracts details
related to the edge structure (orientation, strength, etc) and
simply tries to predict the presence/absence of an edge at
a given point. Due to such abstractions, the classiﬁcation
branch produces contour predictions with high recall.
The second branch is optimized as a regressor to predict

Accepted to CVPR 2015


a given point. Due to such abstractions, the classiﬁcation
branch produces contour predictions with high recall.
The second branch is optimized as a regressor to predict
the fraction of human labelers agreeing about the contour
presence at a particular point. Due to a regression objective,
this branch is much more selective than the ﬁrst branch. Intuitively, the second branch is trained to learn the structural
differences between the contours that are marked by a different fraction of human labelers. For instance, the area that
was labeled as a contour by 80% of human labelers must
be signiﬁcantly different than the area that was labeled as
a contour by 20% human labelers. The regression branch
is trying to learn such differences by predicting the fraction of human labelers who would mark a particular point
as a contour. Thus, in a sense, we are training the regression branch to implicitly mimic how human labelers reason about the contour presence at a given point. In the experimental section, we demonstrate that due to its selectivity, the regression branch produces contour predictions with
very high precision. In Fig. 4, we present several samples
of ground truth data that illustrate the different properties of
our two end-objectives.
The number of hidden layers in the ﬁrst and second fully

Accepted to CVPR 2015


of ground truth data that illustrate the different properties of
our two end-objectives.
The number of hidden layers in the ﬁrst and second fully
connected layers of both branches are 1024 and512, respectively. Both branches optimize the sum of squared difference loss over the (binary or continuous) labels. At test-ing time, the scalar outputs computed from these two subnetworks are averaged to produce a ﬁnal score indicative of
the probability that the candidate point is a contour. Visualization of this architecture is presented in Fig. 5.
In order to train our sub-network, we generate patch examples and labels using training images with ground truth
annotations from multiple human labelers. To generate the
binary labels, we ﬁrst sample 40;000positive examples that
were marked as contours by at least one of the labelers. To
generate negative examples we consider the points that were
selected as candidate contour points by the Canny edge detector but that have not been marked as contours by any of
the human labelers. These are essentially false positives.
For training, we use a random subset of 40;000 of such
points in order to have equal proportion of negative and
positive examples. These 80;000examples are then used
to train our classiﬁcation sub-network.

Accepted to CVPR 2015


points in order to have equal proportion of negative and
positive examples. These 80;000examples are then used
to train our classiﬁcation sub-network.
In addition to the binary labels, we also generate continuous labels that are used to train the regression network.
For this purpose, we deﬁne the regression label of a point to
be the fraction of human labelers that marked the point as a
contour. These 80;000examples with continuous labels are
then also used to train our regression sub-network.
3.2. Multi-Scale Architecture
In the previous section we presented a convolutional architecture for contour prediction utilizing a single scale.
However, in practice, we found that a multi-scale approach
works much better. In this section, we show how to modify
the single-scale architecture so that it can exploit multiple
scales simultaneously.
Rather than extracting a patch at a single scale as we did
in the previous section, in a multi-scale setting we extract
|{z}
|{z}Fixed WeightsLearned WeightsClassiﬁcation Branch
Regression Branch
}
}Figure 5: Detailed illustration of our proposed architecture in a single-scale setting. First, an input patch, centered around the

Accepted to CVPR 2015