Regression Branch
}
}Figure 5: Detailed illustration of our proposed architecture in a single-scale setting. First, an input patch, centered around the
candidate point, goes through ﬁve convolutional layers of the KNet . To extract high-level features, at each convolutional layer
we extract a small sub-volume of the feature map around the center point, and perform max,average , and center pooling on
this sub-volume. The pooled values feed a bifurcated sub-network. At testing time, the scalar outputs computed from the
branches of a bifurcated sub-networks are averaged to produce a ﬁnal contour prediction.
patches around the candidate point for different patch sizes
so that they cover different spatial extents of the image. We
then resize the patches to ﬁt the KNet input and pump them
in parallel through the ﬁve convolutional layers. Our highlevel features are then built by performing max, average and
center pooling in a small sub-volume of the feature map at
each convolutional layer and at each scale. This effectively
increases the dimensionality of the feature vector by a factor
equal to the number of scales compared to the single-scale
setting. These pooled features are then connected as before

Accepted to CVPR 2015


increases the dimensionality of the feature vector by a factor
equal to the number of scales compared to the single-scale
setting. These pooled features are then connected as before
to the two separately-trained network branches. A visualization of our multi-scale architecture is shown in Fig. 2.
3.3. Implementation Details
In this section, we describe additional implementation
details of our model. Our deep network is implemented using the software library Caffe [14].
We use four different scales for our patches. The sizes
of these patches are 6464;128128;196196and a
full-sized image. All of the patches are then resized to the
KNet input dimensions of 227227.
When extracting high-level features from the convolutional layers of KNet , we use sub-volumes of convolutionalfeature maps having spatial sizes 77;55;33;33,
and33for the convolutional layers 1;2;3;4;5, respectively. Note that we shrink the size of the subvolume as we
go deeper in the network since the feature maps get smaller
due to pooling. Our choice of subvolume sizes is made to
ensure we are roughly considering the same spatial extent
of the original image at each layer.
As illustrated in Fig. 5, during the training the weights

DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection

Summary:
Contour detection has been a fundamental component in many image segmentation
and object detection systems. Most previous work utilizes low-level features
such as texture or saliency to detect contours and then use them as cues for a
higher-level task such as object detection. However, we claim that recognizing
objects and predicting contours are two mutually related tasks. Contrary to
traditional approaches, we show that we can invert the commonly established
pipeline: instead of detecting contours with low-level cues for a higher-level
recognition task, we exploit object-related features as high-level cues for
contour detection.
  We achieve this goal by means of a multi-scale deep network that consists of
five convolutional layers and a bifurcated fully-connected sub-network. The
section from the input layer to the fifth convolutional layer is fixed and
directly lifted from a pre-trained network optimized over a large-scale object
classification task. This section of the network is applied to four different
scales of the image input. These four parallel and identical streams are then
attached to a bifurcated sub-network consisting of two independently-trained
branches. One branch learns to predict the contour likelihood (with a
classification objective) whereas the other branch is trained to learn the
fraction of human labelers agreeing about the contour presence at a given point
(with a regression criterion).
  We show that without any feature engineering our multi-scale deep learning
approach achieves state-of-the-art results in contour detection.

Gedas Bertasius
Jianbo Shi
Lorenzo Torresani

Accepted to CVPR 2015


ensure we are roughly considering the same spatial extent
of the original image at each layer.
As illustrated in Fig. 5, during the training the weights
in the convolutional layers are ﬁxed and only the weights in
the fully connected layers of the two branches are learned.
To train our model we use the learning rate of 0:1, the
dropout fraction of 0:5,50number of epochs, and the size
of the batch equal to 100.
As described earlier, to train classiﬁcation and regression branches we sample 80;000examples with binary labels. We also generate continuous labels for these 80;000
examples. In addition, we sample a hold-out dataset of size
40;000. This hold-out dataset is used for the hard-positive
mining step [22].
For the ﬁrst 25epochs we train classiﬁcation and regression branches independently on the original 80;000sample training dataset. After the ﬁrst 25epochs, we test both
branches on the hold-out dataset and detect false negative
predictions made by each branch. We then use these false
negative examples along with the same number of randomly
selected true negative examples to augment our original
80;000training dataset. For the remaining 25epochs, we
train both branches on this augmented dataset.

Accepted to CVPR 2015


selected true negative examples to augment our original
80;000training dataset. For the remaining 25epochs, we
train both branches on this augmented dataset.
The motivation for the hard-positive mining step is to reduce the number of false negative predictions produced by
both branches. By augmenting the original 80;000sized
training data with false negative examples, we are forcing
both branches to focus on hard positive examples, and thus,
effectively reducing the number of false negative predictions.
4. Experiments
In this section, we present our results on the BSDS500
dataset [23], which is arguably the most established benchmark for contour detection. This dataset contains 200training images, 100 validation images, and 200 test images.
Contour detection accuracy is evaluated using three standard measures: ﬁxed contour threshold (ODS), per-image
best threshold (OIS), and average precision (AP).
In section 4.1 we quantitatively compare our approach
to the state-of-the-art. In sections 4.2-4.5 we study how the
performance of our system changes as we modify some of
the architecture choices (number of scales, feature maps,
pooling scheme, training objective). This will cast additional insight into the factors that critically contribute to the
high accuracy of our system.
4.1. Comparison with Other Methods

Accepted to CVPR 2015


pooling scheme, training objective). This will cast additional insight into the factors that critically contribute to the
high accuracy of our system.
4.1. Comparison with Other Methods
We compare the results produced by our approach and
previously proposed contour detection methods. Table 1
summarizes the results. We note that our algorithm achieves
contour detection accuracy that is higher or equal to stateof-the-art results according to two of the three metrics.
Fig. 1 shows the precision and recall curve for the methods considered in our comparison. It also lists the F-score
for each method (in the legend). We observe that there is the
accuracy margin separating our approach from prior techniques. In particular, for low-recall our method achieves almost perfect precision rate. It also produces state-of-the-art
F-score.
4.2. Single Scale versus Multiple Scales
In this section we study the beneﬁts of a multi-scale architecture. Results in Table 2 report accuracy for different numbers and choices of scales. The ﬁrst four rows in
the table illustrate the results achieved using a single-scale
approach. Speciﬁcally, these four cases show performance
obtained when training and testing our system with an input

Accepted to CVPR 2015


the table illustrate the results achieved using a single-scale
approach. Speciﬁcally, these four cases show performance
obtained when training and testing our system with an input
patch of size 6464;128128;196196or a full-sized image, respectively. Note that adding information from multiple scales leads to signiﬁcantly higher F-scores and higherMethod ODS OIS AP
Felz., Hutt. [8] 0.610 0.640 0.560
Mean Shift [6] 0.640 0.680 0.560
Ncuts [30] 0.640 0.680 0.450
SCT [20] 0.710 0.720 0.740
gPb-owt-ucm [1] 0.726 0.757 0.696
Sketch Tokens [18] 0.727 0.746 0.780
PMI [13] 0.737 0.771 0.783
DeepNet [16] 0.738 0.759 0.758
SCG [26] 0.739 0.758 0.773
SE [7] 0.746 0.767 0.803
MCG [2] 0.747 0.779 0.759
N4-ﬁelds [9] 0.753 0.769 0.784

Accepted to CVPR 2015


MCG [2] 0.747 0.779 0.759
N4-ﬁelds [9] 0.753 0.769 0.784
DeepEdge 0.753 0.772 0.807
Table 1: Edge detection results on the BSDS500 benchmark. Our DeepEdge method achieves state-of-the-art contour detections results according to both F-score and AP
metrics.
Scale ODS OIS AP
64 0.71 0.73 0.76
128 0.72 0.74 0.78
196 0.71 0.73 0.76
Full Image 0.67 0.69 0.57
64, 128 0.72 0.75 0.78
64, 128,196 0.72 0.75 0.78
64,128,196,Full Image 0.75 0.77 0.81
Table 2: Results illustrating the effect of using a multi-scale
architecture. Considering multiple scales for contour detection yields signiﬁcantly higher accuracy relative to a single
scale approach.
average precisions. Thus, these results suggest that a multiscale approach is highly advantageous in comparison to a
single scale setting.
4.3. Advantages of Higher-Level Features
In this section, we examine the validity of our earlier

Accepted to CVPR 2015


single scale setting.
4.3. Advantages of Higher-Level Features
In this section, we examine the validity of our earlier
claim that higher-level object-features enhance contour detection accuracy. In Table 3, we present individual contour
detection results using features from the different convolutional layers of KNet . Note that the 4thconvolutional layer
produces the most effective features when considering one
layer at a time. From our earlier discussion we know that
the4thconvolutional layer encodes higher-level object information related to shape and speciﬁc object parts. This
indicates that object speciﬁc cues are particularly beneﬁcial
for contour detection accuracy.
We also observe that by incorporating features from all
Input Image
 Canny Edges
 Raw DeepEdges
 Thresholded DeepEdges
 Ground Truth Edges
Figure 6: Qualitative results produced by our method. Notice how our method learns to distinguish between strong and
weak contours. For instance, in the last row of predictions, contours corresponding to zebra stripes are assigned much lower
probabilities than contours that correspond to the actual object boundaries separating the zebras from the background.
Conv. Layers ODS OIS AP
1st0.66 0.68 0.69
2nd0.71 0.74 0.76

Accepted to CVPR 2015


Conv. Layers ODS OIS AP
1st0.66 0.68 0.69
2nd0.71 0.74 0.76
3rd0.74 0.75 0.79
4th0.74 0.76 0.79
5th0.73 0.74 0.77
All 0.75 0.77 0.81
Table 3: This table shows the advantage of using higherlevel features from the KNet convolutional layers. Individually, the 4thconvolutional layer produces the best contour
prediction results, which implies that higher-level object information is indeed beneﬁcial for contour detection. Combining the features from all convolutional layers leads to
state-of-the-art results.
the convolutional layers, our method achieves state-of-theart contour detection results. This suggests that the features
computed by different layers are complementary and that
considering information from the entire hierarchy is advantageous.4.4. Pooling Scheme
When presenting the architecture of our model, we discussed three different types of pooling: max,average , and
center pooling. These three techniques were used to pool
the values from the sub-volumes extracted around the center point in each convolutional ﬁlter as illustrated in Fig. 5.
We now show how each type of pooling affects contour

Accepted to CVPR 2015


We now show how each type of pooling affects contour
detection results. Table 4 illustrates that, individually, center pooling yields the best contour detection results. This is
expected because the candidate point for which we are trying to predict a contour probability is located at the center
of the input patch.
However, we note that combining all three types of pooling, achieves better contour detection results than any single
pooling technique alone.
4.5. Bifurcation and Training Objective
Next, we want to show that that the two independentlytrained classiﬁcation and regression branches in the bifurcated sub-network provide complementary information that
yields improved contour detection accuracy. In Table 5, we
present contour detection results achieved by using predictions from the individual branches of the bifurcated subnetwork.
Pooling Type ODS OIS AP
Average 0.73 0.75 0.78
Max 0.69 0.72 0.73
Center 0.74 0.76 0.8
Avg+Max+Cen 0.75 0.77 0.81
Table 4: Effect of different pooling schemes on contour detection results. Center pooling produces better results than
max or average pooling. Combining all three types of pooling further improves the results.
Branch ODS OIS AP

Accepted to CVPR 2015


max or average pooling. Combining all three types of pooling further improves the results.
Branch ODS OIS AP
Classiﬁcation 0.75 0.76 0.78
Regression 0.74 0.76 0.80
Classiﬁcation+Regression 0.75 0.77 0.81
Table 5: Contour detection accuracy of the two branches
in our bifurcated sub-network. The classiﬁcation branch
yields solid F-score results whereas the regression branch
achieves high average precision. Averaging the outputs
from these two branches further improve the results.
From these results, we observe that using predictions
just from the classiﬁcation branch produces high F-score
whereas using predictions only from the regression branch
yields high average precision. Combining the predictions
from both branches improves the results according to both
metrics thus, supporting our claim that separately optimizing edge classiﬁcation and regression objectives is beneﬁcial to contour detection.
4.6. Qualitative Results
Finally, we present qualitative results produced by our
method. In Figure 6 we show for each input image example, the set of candidate points produced by the Canny edge
detector, the un-thresholded predictions of our method, the

Accepted to CVPR 2015


method. In Figure 6 we show for each input image example, the set of candidate points produced by the Canny edge
detector, the un-thresholded predictions of our method, the
thresholded predictions, and the ground truth contour map
computed as an average of the multiple manual annotations.
To generate the thresholded predictions, we use a probability threshold of 0:5.
Note that our method successfully distinguishes between
strong and weak contours. Speciﬁcally, observe that in the
last row of Figure 6, our method assigns lower probability
to contours corresponding to zebra stripes compared to the
contours of the actual object boundary separating the zebras
from the background. Thus, in the thresholded version of
the prediction, the weak contours inside the zebra bodies
are removed and we obtain contour predictions that look
very similar to the ground truth.
Due to locality of our method, it may be beneﬁcial
to apply spectral methods [21, 30] or conditional randomﬁelds [27] on top of our method to further improve its performance.
4.7. Computational Cost
In its current form, our method requires about 60KKNet
evaluations ( 15Kper scale) to extract the features. Based
on the runtimes reported in [14], if executed on a GPU our

Accepted to CVPR 2015


evaluations ( 15Kper scale) to extract the features. Based
on the runtimes reported in [14], if executed on a GPU our
method would take about 5 minutes and could be made
faster using the approach described in [12].
An alternative way to dramatically reduce the runtime of
DeepEdge is to interpolate the entries of the feature maps
produced by applying the KNet to the full image rather than
to individual patches. Such an approach would reduce the
number of CNN evaluations needed from 60Kto4(one
for each scale), which would allow our method to run in
real time even on CPUs. We note that interpolation of features in deep layers has been used successfully in several
recent vision papers [29, 11, 19]. Thus, we believe that such
an approach could yield nearly equivalent contour detection
accuracy, up to a small possible degradation caused by interpolation.
Since in this work we were primarily interested in studying the effective advantage enabled by object-level features
in contour detection, we have not invested any effort in optimizing the implementation of our method. This will be one
of our immediate goals in the future.
5. Conclusions
In this work, we presented a multi-scale bifurcated deep
network for top-down contour detection. In the past, contour detection has been approached as a bottom-up task

Accepted to CVPR 2015


network for top-down contour detection. In the past, contour detection has been approached as a bottom-up task
where low-level features are engineered ﬁrst, then contour
detection is performed, and ﬁnally contours may be used
as cues for object detection. However, due to a close relationship between object and contour detection tasks, we
proposed to invert this pipeline and perform contour detection in a top-down fashion. We demonstrated how to use
higher-level object cues to predict contours and showed that
considering higher-level object-features leads to a substantial gain in contour detection accuracy.
Additionally, we demonstrated that our multi-scale architecture is beneﬁcial to contour prediction as well. By
considering multiple scales, our method incorporates local
and global information around the candidate contour points,
which leads to signiﬁcantly better contour detection results.
Furthermore, we showed that independent optimization of
contour classiﬁcation and regression objectives improves
contour prediction accuracy as well. As our experiments indicate, DeepEdge achieves higher average precision results
compared to any prior or concurrent work.
In conclusion, our results suggest that pure CNN systems can be applied successfully to contour detection and
possibly to many other low-level vision tasks.
6. Acknowledgements

Accepted to CVPR 2015


In conclusion, our results suggest that pure CNN systems can be applied successfully to contour detection and
possibly to many other low-level vision tasks.
6. Acknowledgements
We thank Piotr Teterwak, Du Tran and Mohammad Haris
Baig for helpful discussions. This research was funded in
part by NSF award CNS-1205521.
References
[1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour
detection and hierarchical image segmentation. IEEE Trans.
Pattern Anal. Mach. Intell. , 33(5):898–916, May 2011. 1, 2,
6
[2] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marqu ´es, and J. Malik. Multiscale combinatorial grouping. In Computer Vision
and Pattern Recognition (CVPR) , 2014. 2, 6
[3] E. Borenstein. Combining top-down and bottom-up segmentation. In In Proceedings IEEE workshop on Perceptual Organization in Computer Vision, CVPR , page 46, 2004. 1
[4] C. J. C. Burges. A tutorial on support vector machines for
pattern recognition. Data Mining and Knowledge Discovery ,
2:121–167, 1998. 2

Accepted to CVPR 2015


[4] C. J. C. Burges. A tutorial on support vector machines for
pattern recognition. Data Mining and Knowledge Discovery ,
2:121–167, 1998. 2
[5] J. Canny. A computational approach to edge detection. IEEE
Trans. Pattern Anal. Mach. Intell. , 8(6):679–698, June 1986.
3
[6] D. Comaniciu, P. Meer, and S. Member. Mean shift: A robust
approach toward feature space analysis. IEEE Transactions
on Pattern Analysis and Machine Intelligence , 24:603–619,
2002. 6
[7] P. Doll ´ar and C. L. Zitnick. Fast edge detection using structured forests. PAMI , 2015. 2, 6
[8] P. F. Felzenszwalb and D. P. Huttenlocher. Efﬁcient
graph-based image segmentation. Int. J. Comput. Vision ,
59(2):167–181, Sept. 2004. 6
[9] Y . Ganin and V . S. Lempitsky. N4-ﬁelds: Neural network
nearest neighbor ﬁelds for image transforms. ACCV , 2014.
2, 6

Accepted to CVPR 2015


nearest neighbor ﬁelds for image transforms. ACCV , 2014.
2, 6
[10] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR) , 2014. 3
[11] B. Hariharan, P. A. Arbel ´aez, R. B. Girshick, and J. Malik.
Hypercolumns for object segmentation and ﬁne-grained localization. CoRR , abs/1411.5752, 2014. 8
[12] F. N. Iandola, M. W. Moskewicz, S. Karayev, R. B. Girshick,
T. Darrell, and K. Keutzer. Densenet: Implementing efﬁcient
convnet descriptor pyramids. CoRR , abs/1404.1869, 2014. 8
[13] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Crisp
boundary detection using pointwise mutual information. In
ECCV , 2014. 2, 6

Accepted to CVPR 2015


boundary detection using pointwise mutual information. In
ECCV , 2014. 2, 6
[14] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint
arXiv:1408.5093 , 2014. 5, 8
[15] S. Karayev, A. Hertzmann, H. Winnemoeller, A. Agarwala, and T. Darrell. Recognizing image style. CoRR ,
abs/1311.3715, 2013. 3[16] J. J. Kivinen, C. K. Williams, N. Heess, and D. Technologies. Visual boundary prediction: A deep neural prediction
network and quality dissection. AISTATS , 1(2):9, 2014. 2, 6
[17] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classiﬁcation with deep convolutional neural networks. In
F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems 25 ,

Accepted to CVPR 2015


F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems 25 ,
pages 1097–1105. Curran Associates, Inc., 2012. 2, 3
[18] J. Lim, C. L. Zitnick, and P. Doll ´ar. Sketch tokens: A learned
mid-level representation for contour and object detection. In
CVPR , 2013. 2, 6
[19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. CoRR , abs/1411.4038,
2014. 8
[20] M. Maire, S. X. Yu, and P. Perona. Reconstructive sparse
code transfer for contour detection and semantic labeling. In
Asian Conference on Computer Vision (ACCV) , 2014. 2, 6
[21] J. Malik, S. Belongie, T. Leung, and J. Shi. Contour and texture analysis for image segmentation. Int. J. Comput. Vision ,
43(1):7–27, June 2001. 8
[22] T. Malisiewicz, A. Gupta, and A. A. Efros. Ensemble of
exemplar-svms for object detection and beyond. In ICCV ,

Accepted to CVPR 2015


exemplar-svms for object detection and beyond. In ICCV ,
2011. 5
[23] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database
of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int’l Conf. Computer Vision , volume 2, pages 416–423, July 2001. 6
[24] A. Opelt and A. Zisserman. A boundary-fragment-model for
object detection. In In ECCV , pages 575–588, 2006. 1
[25] P. H. O. Pinheiro and R. Collobert. Recurrent convolutional
neural networks for scene parsing. CoRR , abs/1306.2795,
2013. 2
[26] X. Ren and L. Bo. Discriminatively Trained Sparse Code
Gradients for Contour Detection. In Advances in Neural Information Processing Systems , December 2012. 2, 6
[27] X. Ren, C. C. Fowlkes, and J. Malik. Scale-invariant contour
completion using condition random ﬁelds. Technical report.
8

Accepted to CVPR 2015


completion using condition random ﬁelds. Technical report.
8
[28] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual
Recognition Challenge, 2014. 3
[29] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR ,
abs/1312.6229, 2013. 8
[30] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine
Intelligence , 22:888–905, 1997. 2, 6, 8
[31] J. Shotton. Contour-based learning for object detection. In
In Proc. ICCV , pages 503–510, 2005. 1
[32] Y . Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface:

Accepted to CVPR 2015


[32] Y . Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface:
Closing the gap to human-level performance in face veriﬁcation. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2014. 2
[33] A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. CoRR , abs/1312.4659, 2013.
2
[34] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev.
Panda: Pose aligned networks for deep attribute modeling.
CoRR , abs/1311.5591, 2013. 3

Accepted to CVPR 2015


Published as a conference paper at ICLR 2015
SEMANTIC IMAGE SEGMENTATION WITH DEEP CONVOLUTIONAL NETS AND FULLY CONNECTED CRF S
Liang-Chieh Chen
Univ. of California, Los Angeles
lcchen@cs.ucla.edu
George Papandreou
Google Inc.
gpapan@google.com
Iasonas Kokkinos
CentraleSup ´elec and INRIA
iasonas.kokkinos@ecp.fr
Kevin Murphy
Google Inc.
kpmurphy@google.com
Alan L. Yuille
Univ. of California, Los Angeles
yuille@stat.ucla.edu
ABSTRACT
Deep Convolutional Neural Networks (DCNNs) have recently shown state of the
art performance in high level vision tasks, such as image classiﬁcation and object detection. This work brings together methods from DCNNs and probabilistic
graphical models for addressing the task of pixel-level classiﬁcation (also called
”semantic image segmentation”). We show that responses at the ﬁnal layer of
DCNNs are not sufﬁciently localized for accurate object segmentation. This is

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Summary:
Deep Convolutional Neural Networks (DCNNs) have recently shown state of the
art performance in high level vision tasks, such as image classification and
object detection. This work brings together methods from DCNNs and
probabilistic graphical models for addressing the task of pixel-level
classification (also called "semantic image segmentation"). We show that
responses at the final layer of DCNNs are not sufficiently localized for
accurate object segmentation. This is due to the very invariance properties
that make DCNNs good for high level tasks. We overcome this poor localization
property of deep networks by combining the responses at the final DCNN layer
with a fully connected Conditional Random Field (CRF). Qualitatively, our
"DeepLab" system is able to localize segment boundaries at a level of accuracy
which is beyond previous methods. Quantitatively, our method sets the new
state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching
71.6% IOU accuracy in the test set. We show how these results can be obtained
efficiently: Careful network re-purposing and a novel application of the 'hole'
algorithm from the wavelet community allow dense computation of neural net
responses at 8 frames per second on a modern GPU.

Liang-Chieh Chen
George Papandreou
Iasonas Kokkinos
Kevin Murphy
Alan L. Yuille

14 pages. Updated related work


DCNNs are not sufﬁciently localized for accurate object segmentation. This is
due to the very invariance properties that make DCNNs good for high level tasks.
We overcome this poor localization property of deep networks by combining the
responses at the ﬁnal DCNN layer with a fully connected Conditional Random
Field (CRF). Qualitatively, our “DeepLab” system is able to localize segment
boundaries at a level of accuracy which is beyond previous methods. Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic
image segmentation task, reaching 71.6% IOU accuracy in the test set. We show
how these results can be obtained efﬁciently: Careful network re-purposing and a
novel application of the ’hole’ algorithm from the wavelet community allow dense
computation of neural net responses at 8 frames per second on a modern GPU.
1 I NTRODUCTION
Deep Convolutional Neural Networks (DCNNs) had been the method of choice for document recognition since LeCun et al. (1998), but have only recently become the mainstream of high-level vision

14 pages. Updated related work


research. Over the past two years DCNNs have pushed the performance of computer vision systems to soaring heights on a broad array of high-level problems, including image classiﬁcation
(Krizhevsky et al., 2013; Sermanet et al., 2013; Simonyan & Zisserman, 2014; Szegedy et al., 2014;
Work initiated when G.P. was with the Toyota Technological Institute at Chicago. The ﬁrst two authors
contributed equally to this work.
1arXiv:1412.7062v4  [cs.CV]  7 Jun 2016
Published as a conference paper at ICLR 2015
Papandreou et al., 2014), object detection (Girshick et al., 2014), ﬁne-grained categorization (Zhang
et al., 2014), among others. A common theme in these works is that DCNNs trained in an end-to-end
manner deliver strikingly better results than systems relying on carefully engineered representations,
such as SIFT or HOG features. This success can be partially attributed to the built-in invariance of
DCNNs to local image transformations, which underpins their ability to learn hierarchical abstractions of data (Zeiler & Fergus, 2014). While this invariance is clearly desirable for high-level vision

14 pages. Updated related work


tasks, it can hamper low-level tasks, such as pose estimation (Chen & Yuille, 2014; Tompson et al.,
2014) and semantic segmentation - where we want precise localization, rather than abstraction of
spatial details.
There are two technical hurdles in the application of DCNNs to image labeling tasks: signal downsampling, and spatial ‘insensitivity’ (invariance). The ﬁrst problem relates to the reduction of signal
resolution incurred by the repeated combination of max-pooling and downsampling (‘striding’) performed at every layer of standard DCNNs (Krizhevsky et al., 2013; Simonyan & Zisserman, 2014;
Szegedy et al., 2014). Instead, as in Papandreou et al. (2014), we employ the ‘atrous’ (with holes)
algorithm originally developed for efﬁciently computing the undecimated discrete wavelet transform
(Mallat, 1999). This allows efﬁcient dense computation of DCNN responses in a scheme substantially simpler than earlier solutions to this problem (Giusti et al., 2013; Sermanet et al., 2013).

14 pages. Updated related work


The second problem relates to the fact that obtaining object-centric decisions from a classiﬁer requires invariance to spatial transformations, inherently limiting the spatial accuracy of the DCNN
model. We boost our model’s ability to capture ﬁne details by employing a fully-connected Conditional Random Field (CRF). Conditional Random Fields have been broadly used in semantic segmentation to combine class scores computed by multi-way classiﬁers with the low-level information
captured by the local interactions of pixels and edges (Rother et al., 2004; Shotton et al., 2009) or
superpixels (Lucchi et al., 2011). Even though works of increased sophistication have been proposed
to model the hierarchical dependency (He et al., 2004; Ladicky et al., 2009; Lempitsky et al., 2011)
and/or high-order dependencies of segments (Delong et al., 2012; Gonfaus et al., 2010; Kohli et al.,
2009; Chen et al., 2013; Wang et al., 2015), we use the fully connected pairwise CRF proposed by
Kr¨ahenb ¨uhl & Koltun (2011) for its efﬁcient computation, and ability to capture ﬁne edge details

14 pages. Updated related work


Kr¨ahenb ¨uhl & Koltun (2011) for its efﬁcient computation, and ability to capture ﬁne edge details
while also catering for long range dependencies. That model was shown in Kr ¨ahenb ¨uhl & Koltun
(2011) to largely improve the performance of a boosting-based pixel-level classiﬁer, and in our work
we demonstrate that it leads to state-of-the-art results when coupled with a DCNN-based pixel-level
classiﬁer.
The three main advantages of our “DeepLab” system are (i) speed: by virtue of the ‘atrous’ algorithm, our dense DCNN operates at 8 fps, while Mean Field Inference for the fully-connected CRF
requires 0.5 second, (ii) accuracy: we obtain state-of-the-art results on the PASCAL semantic segmentation challenge, outperforming the second-best approach of Mostajabi et al. (2014) by a margin
of 7.2 %and (iii) simplicity: our system is composed of a cascade of two fairly well-established modules, DCNNs and CRFs.
2 R ELATED WORK

14 pages. Updated related work


of 7.2 %and (iii) simplicity: our system is composed of a cascade of two fairly well-established modules, DCNNs and CRFs.
2 R ELATED WORK
Our system works directly on the pixel representation, similarly to Long et al. (2014). This is in contrast to the two-stage approaches that are now most common in semantic segmentation with DCNNs:
such techniques typically use a cascade of bottom-up image segmentation and DCNN-based region
classiﬁcation, which makes the system commit to potential errors of the front-end segmentation system. For instance, the bounding box proposals and masked regions delivered by (Arbel ´aez et al.,
2014; Uijlings et al., 2013) are used in Girshick et al. (2014) and (Hariharan et al., 2014b) as inputs
to a DCNN to introduce shape information into the classiﬁcation process. Similarly, the authors of
Mostajabi et al. (2014) rely on a superpixel representation. A celebrated non-DCNN precursor to
these works is the second order pooling method of (Carreira et al., 2012) which also assigns labels
to the regions proposals delivered by (Carreira & Sminchisescu, 2012). Understanding the perils

14 pages. Updated related work


to the regions proposals delivered by (Carreira & Sminchisescu, 2012). Understanding the perils
of committing to a single segmentation, the authors of Cogswell et al. (2014) build on (Yadollahpour et al., 2013) to explore a diverse set of CRF-based segmentation proposals, computed also by
(Carreira & Sminchisescu, 2012). These segmentation proposals are then re-ranked according to a
DCNN trained in particular for this reranking task. Even though this approach explicitly tries to
handle the temperamental nature of a front-end segmentation algorithm, there is still no explicit ex2
Published as a conference paper at ICLR 2015
ploitation of the DCNN scores in the CRF-based segmentation algorithm: the DCNN is only applied
post-hoc, while it would make sense to directly try to use its results during segmentation.
Moving towards works that lie closer to our approach, several other researchers have considered
the use of convolutionally computed DCNN features for dense image labeling. Among the ﬁrst
have been Farabet et al. (2013) who apply DCNNs at multiple image resolutions and then employ a
segmentation tree to smooth the prediction results; more recently, Hariharan et al. (2014a) propose to

14 pages. Updated related work


segmentation tree to smooth the prediction results; more recently, Hariharan et al. (2014a) propose to
concatenate the computed inter-mediate feature maps within the DCNNs for pixel classiﬁcation, and
Dai et al. (2014) propose to pool the inter-mediate feature maps by region proposals. Even though
these works still employ segmentation algorithms that are decoupled from the DCNN classiﬁer’s
results, we believe it is advantageous that segmentation is only used at a later stage, avoiding the
commitment to premature decisions.
More recently, the segmentation-free techniques of (Long et al., 2014; Eigen & Fergus, 2014) directly apply DCNNs to the whole image in a sliding window fashion, replacing the last fully connected layers of a DCNN by convolutional layers. In order to deal with the spatial localization
issues outlined in the beginning of the introduction, Long et al. (2014) upsample and concatenate
the scores from inter-mediate feature maps, while Eigen & Fergus (2014) reﬁne the prediction result
from coarse to ﬁne by propagating the coarse results to another DCNN.
The main difference between our model and other state-of-the-art models is the combination of

14 pages. Updated related work


from coarse to ﬁne by propagating the coarse results to another DCNN.
The main difference between our model and other state-of-the-art models is the combination of
pixel-level CRFs and DCNN-based ‘unary terms’. Focusing on the closest works in this direction,
Cogswell et al. (2014) use CRFs as a proposal mechanism for a DCNN-based reranking system,
while Farabet et al. (2013) treat superpixels as nodes for a local pairwise CRF and use graph-cuts for
discrete inference; as such their results can be limited by errors in superpixel computations, while ignoring long-range superpixel dependencies. Our approach instead treats every pixel as a CRF node,
exploits long-range dependencies, and uses CRF inference to directly optimize a DCNN-driven cost
function. We note that mean ﬁeld had been extensively studied for traditional image segmentation/edge detection tasks, e.g., (Geiger & Girosi, 1991; Geiger & Yuille, 1991; Kokkinos et al.,
2008), but recently Kr ¨ahenb ¨uhl & Koltun (2011) showed that the inference can be very efﬁcient for
fully connected CRF and particularly effective in the context of semantic segmentation.

14 pages. Updated related work


fully connected CRF and particularly effective in the context of semantic segmentation.
After the ﬁrst version of our manuscript was made publicly available, it came to our attention that
two other groups have independently and concurrently pursued a very similar direction, combining
DCNNs and densely connected CRFs (Bell et al., 2014; Zheng et al., 2015). There are several
differences in technical aspects of the respective models. Bell et al. (2014) focus on the problem
of material classiﬁcation, while Zheng et al. (2015) unroll the CRF mean-ﬁeld inference steps to
convert the whole system into an end-to-end trainable feed-forward network.
We have updated our proposed “DeepLab” system with much improved methods and results in our
latest work (Chen et al., 2016). We refer the interested reader to the paper for details.
3 C ONVOLUTIONAL NEURAL NETWORKS FOR DENSE IMAGE LABELING
Herein we describe how we have re-purposed and ﬁnetuned the publicly available Imagenetpretrained state-of-art 16-layer classiﬁcation network of (Simonyan & Zisserman, 2014) (VGG-16)
into an efﬁcient and effective dense feature extractor for our dense semantic image segmentation

14 pages. Updated related work


into an efﬁcient and effective dense feature extractor for our dense semantic image segmentation
system.
3.1 E FFICIENT DENSE SLIDING WINDOW FEATURE EXTRACTION WITH THE HOLE
ALGORITHM
Dense spatial score evaluation is instrumental in the success of our dense CNN feature extractor. As
a ﬁrst step to implement this, we convert the fully-connected layers of VGG-16 into convolutional
ones and run the network in a convolutional fashion on the image at its original resolution. However
this is not enough as it yields very sparsely computed detection scores (with a stride of 32 pixels). To
compute scores more densely at our target stride of 8 pixels, we develop a variation of the method
previously employed by Giusti et al. (2013); Sermanet et al. (2013). We skip subsampling after
the last two max-pooling layers in the network of Simonyan & Zisserman (2014) and modify the
convolutional ﬁlters in the layers that follow them by introducing zeros to increase their length ( 2in
3
Published as a conference paper at ICLR 2015
Input stride
Output stride
Figure 1: Illustration of the hole algorithm in 1-D, when kernel size = 3 ,input stride = 2 , and

14 pages. Updated related work


Input stride
Output stride
Figure 1: Illustration of the hole algorithm in 1-D, when kernel size = 3 ,input stride = 2 , and
output stride = 1 .
the last three convolutional layers and 4in the ﬁrst fully connected layer). We can implement this
more efﬁciently by keeping the ﬁlters intact and instead sparsely sample the feature maps on which
they are applied on using an input stride of 2 or 4 pixels, respectively. This approach, illustrated
in Fig. 1 is known as the ‘hole algorithm’ (‘atrous algorithm’) and has been developed before for
efﬁcient computation of the undecimated wavelet transform (Mallat, 1999). We have implemented
this within the Caffe framework (Jia et al., 2014) by adding to the im2col function (it converts multichannel feature maps to vectorized patches) the option to sparsely sample the underlying feature
map. This approach is generally applicable and allows us to efﬁciently compute dense CNN feature
maps at any target subsampling rate without introducing any approximations.
We ﬁnetune the model weights of the Imagenet-pretrained VGG-16 network to adapt it to the image

14 pages. Updated related work


We ﬁnetune the model weights of the Imagenet-pretrained VGG-16 network to adapt it to the image
classiﬁcation task in a straightforward fashion, following the procedure of Long et al. (2014). We
replace the 1000-way Imagenet classiﬁer in the last layer of VGG-16 with a 21-way one. Our
loss function is the sum of cross-entropy terms for each spatial position in the CNN output map
(subsampled by 8 compared to the original image). All positions and labels are equally weighted in
the overall loss function. Our targets are the ground truth labels (subsampled by 8). We optimize the
objective function with respect to the weights at all network layers by the standard SGD procedure
of Krizhevsky et al. (2013).
During testing, we need class score maps at the original image resolution. As illustrated in Figure 2
and further elaborated in Section 4.1, the class score maps (corresponding to log-probabilities) are
quite smooth, which allows us to use simple bilinear interpolation to increase their resolution by a
factor of 8 at a negligible computational cost. Note that the method of Long et al. (2014) does not
use the hole algorithm and produces very coarse scores (subsampled by a factor of 32) at the CNN

14 pages. Updated related work


use the hole algorithm and produces very coarse scores (subsampled by a factor of 32) at the CNN
output. This forced them to use learned upsampling layers, signiﬁcantly increasing the complexity
and training time of their system: Fine-tuning our network on PASCAL VOC 2012 takes about 10
hours, while they report a training time of several days (both timings on a modern GPU).
3.2 C ONTROLLING THE RECEPTIVE FIELD SIZE AND ACCELERATING DENSE
COMPUTATION WITH CONVOLUTIONAL NETS
Another key ingredient in re-purposing our network for dense score computation is explicitly controlling the network’s receptive ﬁeld size. Most recent DCNN-based image recognition methods
rely on networks pre-trained on the Imagenet large-scale classiﬁcation task. These networks typically have large receptive ﬁeld size: in the case of the VGG-16 net we consider, its receptive ﬁeld
is224224(with zero-padding) and 404404pixels if the net is applied convolutionally. After

14 pages. Updated related work


is224224(with zero-padding) and 404404pixels if the net is applied convolutionally. After
converting the network to a fully convolutional one, the ﬁrst fully connected layer has 4,096 ﬁlters of large 77spatial size and becomes the computational bottleneck in our dense score map
computation.
We have addressed this practical problem by spatially subsampling (by simple decimation) the ﬁrst
FC layer to 44(or33) spatial size. This has reduced the receptive ﬁeld of the network down to
128128(with zero-padding) or 308308(in convolutional mode) and has reduced computation time
for the ﬁrst FC layer by 23times. Using our Caffe-based implementation and a Titan GPU, the
resulting VGG-derived network is very efﬁcient: Given a 306306input image, it produces 3939
4
Published as a conference paper at ICLR 2015
dense raw feature scores at the top of the network at a rate of about 8 frames/sec during testing. The
speed during training is 3 frames/sec. We have also successfully experimented with reducing the
number of channels at the fully connected layers from 4,096 down to 1,024, considerably further

14 pages. Updated related work


speed during training is 3 frames/sec. We have also successfully experimented with reducing the
number of channels at the fully connected layers from 4,096 down to 1,024, considerably further
decreasing computation time and memory footprint without sacriﬁcing performance, as detailed in
Section 5. Using smaller networks such as Krizhevsky et al. (2013) could allow video-rate test-time
dense feature computation even on light-weight GPUs.
4 D ETAILED BOUNDARY RECOVERY : FULLY -CONNECTED CONDITIONAL
RANDOM FIELDS AND MULTI -SCALE PREDICTION
4.1 D EEPCONVOLUTIONAL NETWORKS AND THE LOCALIZATION CHALLENGE
As illustrated in Figure 2, DCNN score maps can reliably predict the presence and rough position
of objects in an image but are less well suited for pin-pointing their exact outline. There is a natural
trade-off between classiﬁcation accuracy and localization accuracy with convolutional networks:
Deeper models with multiple max-pooling layers have proven most successful in classiﬁcation tasks,
however their increased invariance and large receptive ﬁelds make the problem of inferring position
from the scores at their top output levels more challenging.

14 pages. Updated related work


however their increased invariance and large receptive ﬁelds make the problem of inferring position
from the scores at their top output levels more challenging.
Recent work has pursued two directions to address this localization challenge. The ﬁrst approach is
to harness information from multiple layers in the convolutional network in order to better estimate
the object boundaries (Long et al., 2014; Eigen & Fergus, 2014). The second approach is to employ
a super-pixel representation, essentially delegating the localization task to a low-level segmentation
method. This route is followed by the very successful recent method of Mostajabi et al. (2014).
In Section 4.2, we pursue a novel alternative direction based on coupling the recognition capacity
of DCNNs and the ﬁne-grained localization accuracy of fully connected CRFs and show that it is
remarkably successful in addressing the localization challenge, producing accurate semantic segmentation results and recovering object boundaries at a level of detail that is well beyond the reach
of existing methods.
4.2 F ULLY -CONNECTED CONDITIONAL RANDOM FIELDS FOR ACCURATE LOCALIZATION
Image/G.T. DCNN output CRF Iteration 1 CRF Iteration 2 CRF Iteration 10

14 pages. Updated related work


Image/G.T. DCNN output CRF Iteration 1 CRF Iteration 2 CRF Iteration 10
Figure 2: Score map (input before softmax function) and belief map (output of softmax function) for
Aeroplane. We show the score (1st row) and belief (2nd row) maps after each mean ﬁeld iteration.
The output of last DCNN layer is used as input to the mean ﬁeld inference. Best viewed in color.
Traditionally, conditional random ﬁelds (CRFs) have been employed to smooth noisy segmentation
maps (Rother et al., 2004; Kohli et al., 2009). Typically these models contain energy terms that
couple neighboring nodes, favoring same-label assignments to spatially proximal pixels. Qualitatively, the primary function of these short-range CRFs has been to clean up the spurious predictions
of weak classiﬁers built on top of local hand-engineered features.
Compared to these weaker classiﬁers, modern DCNN architectures such as the one we use in this
work produce score maps and semantic label predictions which are qualitatively different. As illustrated in Figure 2, the score maps are typically quite smooth and produce homogeneous classiﬁcation

14 pages. Updated related work


work produce score maps and semantic label predictions which are qualitatively different. As illustrated in Figure 2, the score maps are typically quite smooth and produce homogeneous classiﬁcation
results. In this regime, using short-range CRFs can be detrimental, as our goal should be to recover
detailed local structure rather than further smooth it. Using contrast-sensitive potentials (Rother
5
Published as a conference paper at ICLR 2015
Deep
Convolutional
Neural
Network
InputAeroplane
Coarse Score map
Bi-linear Interpolation Fully Connected CRF Final Output
Figure 3: Model Illustration. The coarse score map from Deep Convolutional Neural Network (with
fully convolutional layers) is upsampled by bi-linear interpolation. A fully connected CRF is applied
to reﬁne the segmentation result. Best viewed in color.
et al., 2004) in conjunction to local-range CRFs can potentially improve localization but still miss
thin-structures and typically requires solving an expensive discrete optimization problem.
To overcome these limitations of short-range CRFs, we integrate into our system the fully connected
CRF model of Kr ¨ahenb ¨uhl & Koltun (2011). The model employs the energy function
E(x) =X
ii(xi) +X

14 pages. Updated related work


E(x) =X
ii(xi) +X
ijij(xi;xj) (1)
where xis the label assignment for pixels. We use as unary potential i(xi) =logP(xi), where
P(xi)is the label assignment probability at pixel ias computed by DCNN. The pairwise potential
isij(xi;xj) =(xi;xj)PK
m=1wmkm(fi;fj), where(xi;xj) = 1 ifxi6=xj, and zero
otherwise ( i.e., Potts Model). There is one pairwise term for each pair of pixels iandjin the image
no matter how far from each other they lie, i.e. the model’s factor graph is fully connected. Each km
is the Gaussian kernel depends on features (denoted as f) extracted for pixel iandjand is weighted
by parameter wm. We adopt bilateral position and color terms, speciﬁcally, the kernels are
w1exp
jjpipjjj2
22jjIiIjjj2
22

+w2exp
jjpipjjj2
22

(2)

14 pages. Updated related work


22

+w2exp
jjpipjjj2
22

(2)
where the ﬁrst kernel depends on both pixel positions (denoted as p) and pixel color intensities
(denoted asI), and the second kernel only depends on pixel positions. The hyper parameters ,
and
control the “scale” of the Gaussian kernels.
Crucially, this model is amenable to efﬁcient approximate probabilistic inference (Kr ¨ahenb ¨uhl &
Koltun, 2011). The message passing updates under a fully decomposable mean ﬁeld approximationb(x) =Q
ibi(xi)can be expressed as convolutions with a Gaussian kernel in feature space.
High-dimensional ﬁltering algorithms (Adams et al., 2010) signiﬁcantly speed-up this computation
resulting in an algorithm that is very fast in practice, less that 0.5 sec on average for Pascal VOC
images using the publicly available implementation of (Kr ¨ahenb ¨uhl & Koltun, 2011).
4.3 M ULTI -SCALE PREDICTION

14 pages. Updated related work


images using the publicly available implementation of (Kr ¨ahenb ¨uhl & Koltun, 2011).
4.3 M ULTI -SCALE PREDICTION
Following the promising recent results of (Hariharan et al., 2014a; Long et al., 2014) we have also
explored a multi-scale prediction method to increase the boundary localization accuracy. Specifically, we attach to the input image and the output of each of the ﬁrst four max pooling layers a
two-layer MLP (ﬁrst layer: 128 3x3 convolutional ﬁlters, second layer: 128 1x1 convolutional ﬁlters) whose feature map is concatenated to the main network’s last layer feature map. The aggregate
feature map fed into the softmax layer is thus enhanced by 5 * 128 = 640 channels. We only adjust
the newly added weights, keeping the other network parameters to the values learned by the method
of Section 3. As discussed in the experimental section, introducing these extra direct connections
from ﬁne-resolution layers improves localization performance, yet the effect is not as dramatic as
the one obtained with the fully-connected CRF.
6
Published as a conference paper at ICLR 2015
Method mean IOU (%)
DeepLab 59.80
DeepLab-CRF 63.74

14 pages. Updated related work


6
Published as a conference paper at ICLR 2015
Method mean IOU (%)
DeepLab 59.80
DeepLab-CRF 63.74
DeepLab-MSc 61.30
DeepLab-MSc-CRF 65.21
DeepLab-7x7 64.38
DeepLab-CRF-7x7 67.64
DeepLab-LargeFOV 62.25
DeepLab-CRF-LargeFOV 67.64
DeepLab-MSc-LargeFOV 64.21
DeepLab-MSc-CRF-LargeFOV 68.70Method mean IOU (%)
MSRA-CFM 61.8
FCN-8s 62.2
TTI-Zoomout-16 64.4
DeepLab-CRF 66.4
DeepLab-MSc-CRF 67.1
DeepLab-CRF-7x7 70.3
DeepLab-CRF-LargeFOV 70.3
DeepLab-MSc-CRF-LargeFOV 71.6
(a) (b)
Table 1: (a) Performance of our proposed models on the PASCAL VOC 2012 ‘val’ set (with training
in the augmented ‘train’ set). The best performance is achieved by exploiting both multi-scale

14 pages. Updated related work


in the augmented ‘train’ set). The best performance is achieved by exploiting both multi-scale
features and large ﬁeld-of-view. (b) Performance of our proposed models (with training in the
augmented ‘trainval’ set) compared to other state-of-art methods on the PASCAL VOC 2012 ‘test’
set.
5 E XPERIMENTAL EVALUATION
Dataset We test our DeepLab model on the PASCAL VOC 2012 segmentation benchmark (Everingham et al., 2014), consisting of 20 foreground object classes and one background class. The
original dataset contains 1;464,1;449, and 1;456images for training, validation, and testing, respectively. The dataset is augmented by the extra annotations provided by Hariharan et al. (2011),
resulting in 10;582training images. The performance is measured in terms of pixel intersectionover-union (IOU) averaged across the 21 classes.
Training We adopt the simplest form of piecewise training, decoupling the DCNN and CRF training stages, assuming the unary terms provided by the DCNN are ﬁxed during CRF training.
For DCNN training we employ the VGG-16 network which has been pre-trained on ImageNet. We

14 pages. Updated related work


For DCNN training we employ the VGG-16 network which has been pre-trained on ImageNet. We
ﬁne-tuned the VGG-16 network on the VOC 21-way pixel-classiﬁcation task by stochastic gradient
descent on the cross-entropy loss function, as described in Section 3.1. We use a mini-batch of 20
images and initial learning rate of 0:001(0:01for the ﬁnal classiﬁer layer), multiplying the learning
rate by 0.1 at every 2000 iterations. We use momentum of 0:9and a weight decay of 0:0005 .
After the DCNN has been ﬁne-tuned, we cross-validate the parameters of the fully connected CRF
model in Eq. (2) along the lines of Kr ¨ahenb ¨uhl & Koltun (2011). We use the default values of
w2= 3 and
= 3 and we search for the best values of w1,, andby cross-validation on a
small subset of the validation set (we use 100 images). We employ coarse-to-ﬁne search scheme.

14 pages. Updated related work


small subset of the validation set (we use 100 images). We employ coarse-to-ﬁne search scheme.
Speciﬁcally, the initial search range of the parameters are w12[5;10],2[50 : 10 : 100] and
2[3 : 1 : 10] (MATLAB notation), and then we reﬁne the search step sizes around the ﬁrst
round’s best values. We ﬁx the number of mean ﬁeld iterations to 10 for all reported experiments.
Evaluation on Validation set We conduct the majority of our evaluations on the PASCAL ‘val’
set, training our model on the augmented PASCAL ‘train’ set. As shown in Tab. 1 (a), incorporating
the fully connected CRF to our model (denoted by DeepLab-CRF) yields a substantial performance
boost, about 4% improvement over DeepLab. We note that the work of Kr ¨ahenb ¨uhl & Koltun
(2011) improved the 27:6%result of TextonBoost (Shotton et al., 2009) to 29:1%, which makes the
improvement we report here (from 59:8%to63:7%) all the more impressive.

14 pages. Updated related work


improvement we report here (from 59:8%to63:7%) all the more impressive.
Turning to qualitative results, we provide visual comparisons between DeepLab and DeepLab-CRF
in Fig. 7. Employing a fully connected CRF signiﬁcantly improves the results, allowing the model
to accurately capture intricate object boundaries.
Multi-Scale features We also exploit the features from the intermediate layers, similar to Hariharan et al. (2014a); Long et al. (2014). As shown in Tab. 1 (a), adding the multi-scale features to our
7
Published as a conference paper at ICLR 2015
Method kernel size input stride receptive ﬁeld # parameters mean IOU (%) Training speed (img/sec)
DeepLab-CRF-7x7 77 4 224 134.3M 67.64 1.44
DeepLab-CRF 44 4 128 65.1M 63.74 2.90
DeepLab-CRF-4x4 44 8 224 65.1M 67.14 2.90
DeepLab-CRF-LargeFOV 33 12 224 20.5M 67.64 4.84
Table 2: Effect of Field-Of-View. We show the performance (after CRF) and training speed on the

14 pages. Updated related work


Table 2: Effect of Field-Of-View. We show the performance (after CRF) and training speed on the
PASCAL VOC 2012 ‘val’ set as the function of (1) the kernel size of ﬁrst fully connected layer, (2)
the input stride value employed in the atrous algorithm.
DeepLab model (denoted as DeepLab-MSc) improves about 1:5%performance, and further incorporating the fully connected CRF (denoted as DeepLab-MSc-CRF) yields about 4% improvement.
The qualitative comparisons between DeepLab and DeepLab-MSc are shown in Fig. 4. Leveraging
the multi-scale features can slightly reﬁne the object boundaries.
Field of View The ‘atrous algorithm’ we employed allows us to arbitrarily control the Field-ofView (FOV) of the models by adjusting the input stride, as illustrated in Fig. 1. In Tab. 2, we
experiment with several kernel sizes and input strides at the ﬁrst fully connected layer. The method,
DeepLab-CRF-7x7, is the direct modiﬁcation from VGG-16 net, where the kernel size = 77and
input stride = 4. This model yields performance of 67:64% on the ‘val’ set, but it is relatively slow

14 pages. Updated related work


input stride = 4. This model yields performance of 67:64% on the ‘val’ set, but it is relatively slow
(1:44images per second during training). We have improved model speed to 2:9images per second
by reducing the kernel size to 44. We have experimented with two such network variants with
different FOV sizes, DeepLab-CRF and DeepLab-CRF-4x4; the latter has large FOV ( i.e., large
input stride) and attains better performance. Finally, we employ kernel size 33and input stride =
12, and further change the ﬁlter sizes from 4096 to 1024 for the last two layers. Interestingly, the
resulting model, DeepLab-CRF-LargeFOV , matches the performance of the expensive DeepLabCRF-7x7. At the same time, it is 3:36times faster to run and has signiﬁcantly fewer parameters
(20.5M instead of 134.3M).
The performance of several model variants is summarized in Tab. 1, showing the beneﬁt of exploiting
multi-scale features and large FOV .
Figure 4: Incorporating multi-scale features improves the boundary segmentation. We show the

14 pages. Updated related work


multi-scale features and large FOV .
Figure 4: Incorporating multi-scale features improves the boundary segmentation. We show the
results obtained by DeepLab and DeepLab-MSc in the ﬁrst and second row, respectively. Best
viewed in color.
Mean Pixel IOU along Object Boundaries To quantify the accuracy of the proposed model near
object boundaries, we evaluate the segmentation accuracy with an experiment similar to Kohli et al.
(2009); Kr ¨ahenb ¨uhl & Koltun (2011). Speciﬁcally, we use the ‘void’ label annotated in val set,
which usually occurs around object boundaries. We compute the mean IOU for those pixels that
are located within a narrow band (called trimap) of ‘void’ labels. As shown in Fig. 5, exploiting
the multi-scale features from the intermediate layers and reﬁning the segmentation results by a fully
connected CRF signiﬁcantly improve the results around object boundaries.
Comparison with State-of-art In Fig. 6, we qualitatively compare our proposed model, DeepLabCRF, with two state-of-art models: FCN-8s (Long et al., 2014) and TTI-Zoomout-16 (Mostajabi

14 pages. Updated related work


et al., 2014) on the ‘val’ set (the results are extracted from their papers). Our model is able to
capture the intricate object boundaries.
8
Published as a conference paper at ICLR 2015
0 510 15 20 25 30 35 405560657075808590Pixelwise Accuracy (%)
Trimap Width (pixels)
DL−MSc−CRF
DeepLab−CRF
DeepLab−MSc
DeepLab
0 510 15 20 25 30 35 4035404550556065mean IOU (%)
Trimap Width (pixels)
DL−MSc−CRF
DeepLab−CRF
DeepLab−MSc
DeepLab
(a) (b) (c)
Figure 5: (a) Some trimap examples (top-left: image. top-right: ground-truth. bottom-left: trimap
of 2 pixels. bottom-right: trimap of 10 pixels). Quality of segmentation result within a band around
the object boundaries for the proposed methods. (b) Pixelwise accuracy. (c) Pixel mean IOU.
(a) FCN-8s vs. DeepLab-CRF (b) TTI-Zoomout-16 vs. DeepLab-CRF

14 pages. Updated related work


(a) FCN-8s vs. DeepLab-CRF (b) TTI-Zoomout-16 vs. DeepLab-CRF
Figure 6: Comparisons with state-of-the-art models on the val set. First row: images. Second row:
ground truths. Third row: other recent models (Left: FCN-8s, Right: TTI-Zoomout-16). Fourth
row: our DeepLab-CRF. Best viewed in color.
Reproducibility We have implemented the proposed methods by extending the excellent Caffe
framework (Jia et al., 2014). We share our source code, conﬁguration ﬁles, and trained models that
allow reproducing the results in this paper at a companion web site https://bitbucket.org/
deeplab/deeplab-public .
Test set results Having set our model choices on the validation set, we evaluate our model variants
on the PASCAL VOC 2012 ofﬁcial ‘test’ set. As shown in Tab. 3, our DeepLab-CRF and DeepLabMSc-CRF models achieve performance of 66:4%and67:1%mean IOU1, respectively. Our models

14 pages. Updated related work


outperform all the other state-of-the-art models (speciﬁcally, TTI-Zoomout-16 (Mostajabi et al.,
2014), FCN-8s (Long et al., 2014), and MSRA-CFM (Dai et al., 2014)). When we increase the FOV
of the models, DeepLab-CRF-LargeFOV yields performance of 70:3%, the same as DeepLab-CRF7x7, while its training speed is faster. Furthermore, our best model, DeepLab-MSc-CRF-LargeFOV ,
attains the best performance of 71:6%by employing both multi-scale features and large FOV .
1http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?
challengeid=11&compid=6
9
Published as a conference paper at ICLR 2015
Figure 7: Visualization results on VOC 2012-val. For each row, we show the input image, the
segmentation result delivered by the DCNN (DeepLab), and the reﬁned segmentation result of the
Fully Connected CRF (DeepLab-CRF). We show our failure modes in the last three rows. Best
viewed in color.
10

14 pages. Updated related work


14 pages. Updated related work


DeepLab-MSc-CRF-LargeFOV 93.1 84.4 54.5 81.5 63.6 65.9 85.1 79.1 83.4 30.7 74.1 59.8 79.0 76.1 83.2 80.8 59.7 82.2 50.4 73.1 63.7 71.6
Table 3: Labeling IOU (%) on the PASCAL VOC 2012 test set, using the trainval set for training.
6 D ISCUSSION
Our work combines ideas from deep convolutional neural networks and fully-connected conditional
random ﬁelds, yielding a novel method able to produce semantically accurate predictions and detailed segmentation maps, while being computationally efﬁcient. Our experimental results show that
the proposed method signiﬁcantly advances the state-of-art in the challenging PASCAL VOC 2012
semantic image segmentation task.
There are multiple aspects in our model that we intend to reﬁne, such as fully integrating its two
main components (CNN and CRF) and train the whole system in an end-to-end fashion, similar to
Kr¨ahenb ¨uhl & Koltun (2013); Chen et al. (2014); Zheng et al. (2015). We also plan to experiment

14 pages. Updated related work


Kr¨ahenb ¨uhl & Koltun (2013); Chen et al. (2014); Zheng et al. (2015). We also plan to experiment
with more datasets and apply our method to other sources of data such as depth maps or videos. Recently, we have pursued model training with weakly supervised annotations, in the form of bounding
boxes or image-level labels (Papandreou et al., 2015).
At a higher level, our work lies in the intersection of convolutional neural networks and probabilistic
graphical models. We plan to further investigate the interplay of these two powerful classes of
methods and explore their synergistic potential for solving challenging computer vision tasks.
ACKNOWLEDGMENTS
This work was partly supported by ARO 62250-CS, NIH Grant 5R01EY022247-03, EU Project
RECONFIG FP7-ICT-600825 and EU Project MOBOT FP7-ICT-2011-600796. We also gratefully
acknowledge the support of NVIDIA Corporation with the donation of GPUs used for this research.
We would like to thank the anonymous reviewers for their detailed comments and constructive feedback.
PAPER REVISIONS
Here we present the list of major paper revisions for the convenience of the readers.
v1 Submission to ICLR 2015. Introduces the model DeepLab-CRF, which attains the performance

14 pages. Updated related work


v1 Submission to ICLR 2015. Introduces the model DeepLab-CRF, which attains the performance
of66:4%on PASCAL VOC 2012 test set.
v2 Rebuttal for ICLR 2015. Adds the model DeepLab-MSc-CRF, which incorporates multi-scale
features from the intermediate layers. DeepLab-MSc-CRF yields the performance of 67:1%on
PASCAL VOC 2012 test set.
v3 Camera-ready for ICLR 2015. Experiments with large Field-Of-View. On PASCAL VOC 2012
test set, DeepLab-CRF-LargeFOV achieves the performance of 70:3%. When exploiting both mutliscale features and large FOV , DeepLab-MSc-CRF-LargeFOV attains the performance of 71:6%.
v4 Reference to our updated “DeepLab” system (Chen et al., 2016) with much improved results.
REFERENCES
Adams, A., Baek, J., and Davis, M. A. Fast high-dimensional ﬁltering using the permutohedral
lattice. In Computer Graphics Forum , 2010.

14 pages. Updated related work


lattice. In Computer Graphics Forum , 2010.
Arbel ´aez, P., Pont-Tuset, J., Barron, J. T., Marques, F., and Malik, J. Multiscale combinatorial
grouping. In CVPR , 2014.
11
Published as a conference paper at ICLR 2015
Bell, S., Upchurch, P., Snavely, N., and Bala, K. Material recognition in the wild with the materials
in context database. arXiv:1412.0623 , 2014.
Carreira, J. and Sminchisescu, C. Cpmc: Automatic object segmentation using constrained parametric min-cuts. PAMI , 2012.
Carreira, J., Caseiro, R., Batista, J., and Sminchisescu, C. Semantic segmentation with second-order
pooling. In ECCV , 2012.
Chen, L.-C., Papandreou, G., and Yuille, A. Learning a dictionary of shape epitomes with applications to image labeling. In ICCV , 2013.
Chen, L.-C., Schwing, A., Yuille, A., and Urtasun, R. Learning deep structured models.
arXiv:1407.2538 , 2014.

14 pages. Updated related work


arXiv:1407.2538 , 2014.
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. Deeplab: Semantic
image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.
arXiv:1606.00915 , 2016.
Chen, X. and Yuille, A. L. Articulated pose estimation by a graphical model with image dependent
pairwise relations. In NIPS , 2014.
Cogswell, M., Lin, X., Purushwalkam, S., and Batra, D. Combining the best of graphical models
and convnets for semantic segmentation. arXiv:1412.4313 , 2014.
Dai, J., He, K., and Sun, J. Convolutional feature masking for joint object and stuff segmentation.
arXiv:1412.1283 , 2014.
Delong, A., Osokin, A., Isack, H. N., and Boykov, Y . Fast approximate energy minimization with
label costs. IJCV , 2012.
Eigen, D. and Fergus, R. Predicting depth, surface normals and semantic labels with a common

14 pages. Updated related work


label costs. IJCV , 2012.
Eigen, D. and Fergus, R. Predicting depth, surface normals and semantic labels with a common
multi-scale convolutional architecture. arXiv:1411.4734 , 2014.
Everingham, M., Eslami, S. M. A., Gool, L. V ., Williams, C. K. I., Winn, J., and Zisserma, A. The
pascal visual object classes challenge a retrospective. IJCV , 2014.
Farabet, C., Couprie, C., Najman, L., and LeCun, Y . Learning hierarchical features for scene labeling.PAMI , 2013.
Geiger, D. and Girosi, F. Parallel and deterministic algorithms from mrfs: Surface reconstruction.
PAMI , 13(5):401–412, 1991.
Geiger, D. and Yuille, A. A common framework for image segmentation. IJCV , 6(3):227–243,
1991.
Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object
detection and semantic segmentation. In CVPR , 2014.

14 pages. Updated related work


detection and semantic segmentation. In CVPR , 2014.
Giusti, A., Ciresan, D., Masci, J., Gambardella, L., and Schmidhuber, J. Fast image scanning with
deep max-pooling convolutional neural networks. In ICIP , 2013.
Gonfaus, J. M., Boix, X., Van de Weijer, J., Bagdanov, A. D., Serrat, J., and Gonzalez, J. Harmony
potentials for joint classiﬁcation and segmentation. In CVPR , 2010.
Hariharan, B., Arbel ´aez, P., Bourdev, L., Maji, S., and Malik, J. Semantic contours from inverse
detectors. In ICCV , 2011.
Hariharan, B., Arbel ´aez, P., Girshick, R., and Malik, J. Hypercolumns for object segmentation and
ﬁne-grained localization. arXiv:1411.5752 , 2014a.
Hariharan, B., Arbel ´aez, P., Girshick, R., and Malik, J. Simultaneous detection and segmentation.
InECCV , 2014b.
12
Published as a conference paper at ICLR 2015

14 pages. Updated related work


InECCV , 2014b.
12
Published as a conference paper at ICLR 2015
He, X., Zemel, R. S., and Carreira-Perpindn, M. Multiscale conditional random ﬁelds for image
labeling. In CVPR , 2004.
Jia, Y ., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell,
T. Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093 , 2014.
Kohli, P., Ladicky, L., and Torr, P. H. Robust higher order potentials for enforcing label consistency.
IJCV , 2009.
Kokkinos, I., Deriche, R., Faugeras, O., and Maragos, P. Computational analysis and learning for a
biologically motivated model of boundary detection. Neurocomputing , 71(10):1798–1812, 2008.
Kr¨ahenb ¨uhl, P. and Koltun, V . Efﬁcient inference in fully connected crfs with gaussian edge potentials. In NIPS , 2011.

14 pages. Updated related work


Kr¨ahenb ¨uhl, P. and Koltun, V . Parameter learning and convergent inference for dense random ﬁelds.
InICML , 2013.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classiﬁcation with deep convolutional
neural networks. In NIPS , 2013.
Ladicky, L., Russell, C., Kohli, P., and Torr, P. H. Associative hierarchical crfs for object class image
segmentation. In ICCV , 2009.
LeCun, Y ., Bottou, L., Bengio, Y ., and Haffner, P. Gradient-based learning applied to document
recognition. In Proc. IEEE , 1998.
Lempitsky, V ., Vedaldi, A., and Zisserman, A. Pylon model for semantic segmentation. In NIPS ,
2011.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation.
arXiv:1411.4038 , 2014.
Lucchi, A., Li, Y ., Boix, X., Smith, K., and Fua, P. Are spatial and global constraints really necessary

14 pages. Updated related work


Lucchi, A., Li, Y ., Boix, X., Smith, K., and Fua, P. Are spatial and global constraints really necessary
for segmentation? In ICCV , 2011.
Mallat, S. A Wavelet Tour of Signal Processing . Acad. Press, 2 edition, 1999.
Mostajabi, M., Yadollahpour, P., and Shakhnarovich, G. Feedforward semantic segmentation with
zoom-out features. arXiv:1412.0774 , 2014.
Papandreou, G., Kokkinos, I., and Savalle, P.-A. Untangling local and global deformations in deep
convolutional networks for image classiﬁcation and sliding window detection. arXiv:1412.0296 ,
2014.
Papandreou, G., Chen, L.-C., Murphy, K., and Yuille, A. L. Weakly- and semi-supervised learning
of a DCNN for semantic image segmentation. arXiv:1502.02734 , 2015.
Rother, C., Kolmogorov, V ., and Blake, A. Grabcut: Interactive foreground extraction using iterated
graph cuts. In SIGGRAPH , 2004.

14 pages. Updated related work


Rother, C., Kolmogorov, V ., and Blake, A. Grabcut: Interactive foreground extraction using iterated
graph cuts. In SIGGRAPH , 2004.
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y . Overfeat: Integrated
recognition, localization and detection using convolutional networks. arXiv:1312.6229 , 2013.
Shotton, J., Winn, J., Rother, C., and Criminisi, A. Textonboost for image understanding: Multiclass object recognition and segmentation by jointly modeling texture, layout, and context. IJCV ,
2009.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 , 2014.
Szegedy, C., Liu, W., Jia, Y ., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V ., and
Rabinovich, A. Going deeper with convolutions. arXiv:1409.4842 , 2014.
13
Published as a conference paper at ICLR 2015

14 pages. Updated related work


Rabinovich, A. Going deeper with convolutions. arXiv:1409.4842 , 2014.
13
Published as a conference paper at ICLR 2015
Tompson, J., Jain, A., LeCun, Y ., and Bregler, C. Joint Training of a Convolutional Network and a
Graphical Model for Human Pose Estimation. In NIPS , 2014.
Uijlings, J., van de Sande, K., Gevers, T., and Smeulders, A. Selective search for object recognition.
IJCV , 2013.
Wang, P., Shen, X., Lin, Z., Cohen, S., Price, B., and Yuille, A. Towards uniﬁed depth and semantic
prediction from a single image. In CVPR , 2015.
Yadollahpour, P., Batra, D., and Shakhnarovich, G. Discriminative re-ranking of diverse segmentations. In CVPR , 2013.
Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In ECCV , 2014.
Zhang, N., Donahue, J., Girshick, R., and Darrell, T. Part-based r-cnns for ﬁne-grained category
detection. In ECCV , 2014.

14 pages. Updated related work


detection. In ECCV , 2014.
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V ., Su, Z., Du, D., Huang, C., and Torr, P.
Conditional random ﬁelds as recurrent neural networks. arXiv:1502.03240 , 2015.
14

14 pages. Updated related work


arXiv:1502.03167v3  [cs.LG]  2 Mar 2015BatchNormalization: AcceleratingDeepNetworkTrainingb y
ReducingInternalCovariateShift
Sergey Ioffe
GoogleInc., sioffe@google.comChristianSzegedy
GoogleInc., szegedy@google.com
Abstract
TrainingDeepNeuralNetworksiscomplicatedbythefact
that the distributionofeach layer’sinputschangesduring
training, as the parametersof the previouslayers change.
This slows down the training by requiringlower learning
ratesandcarefulparameterinitialization,andmakesitno toriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate
shift, and address the problem by normalizing layer inputs. Ourmethoddrawsitsstrengthfrommakingnormalizationapartofthemodelarchitectureandperformingthe
normalization for each training mini-batch . Batch Normalizationallowsustousemuchhigherlearningratesand
be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout.
Applied to a state-of-the-art image classiﬁcation model,
Batch Normalizationachievesthe same accuracy with 14
times fewer training steps, and beats the original model

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Summary:
Training Deep Neural Networks is complicated by the fact that the
distribution of each layer's inputs changes during training, as the parameters
of the previous layers change. This slows down the training by requiring lower
learning rates and careful parameter initialization, and makes it notoriously
hard to train models with saturating nonlinearities. We refer to this
phenomenon as internal covariate shift, and address the problem by normalizing
layer inputs. Our method draws its strength from making normalization a part of
the model architecture and performing the normalization for each training
mini-batch. Batch Normalization allows us to use much higher learning rates and
be less careful about initialization. It also acts as a regularizer, in some
cases eliminating the need for Dropout. Applied to a state-of-the-art image
classification model, Batch Normalization achieves the same accuracy with 14
times fewer training steps, and beats the original model by a significant
margin. Using an ensemble of batch-normalized networks, we improve upon the
best published result on ImageNet classification: reaching 4.9% top-5
validation error (and 4.8% test error), exceeding the accuracy of human raters.

Knowledge Matters: Importance of Prior Information for Optimization


Batch Normalizationachievesthe same accuracy with 14
times fewer training steps, and beats the original model
by a signiﬁcant margin. Using an ensemble of batchnormalizednetworks,weimproveuponthebestpublished
result on ImageNet classiﬁcation: reaching 4.9% top-5
validation error (and 4.8% test error), exceeding the accuracyofhumanraters.
1 Introduction
Deep learning has dramatically advanced the state of the
art in vision, speech, and many other areas. Stochastic gradient descent (SGD) has proved to be an effective way of training deep networks, and SGD variants
such as momentum (Sutskeveret al., 2013) and Adagrad
(Duchiet al.,2011)havebeenusedtoachievestate ofthe
art performance. SGD optimizes the parameters Θof the
network,soasto minimizetheloss
Θ = argmin
Θ1
NN/summationdisplay
i=1ℓ(xi,Θ)


network,soasto minimizetheloss
Θ = argmin
Θ1
NN/summationdisplay
i=1ℓ(xi,Θ)
wherex1...Nisthetrainingdataset. With SGD,thetrainingproceedsinsteps,andateachstepweconsidera minibatchx1...mofsizem. The mini-batchis usedtoapproximate the gradient of the loss functionwith respect to the
parameters,bycomputing
1
m∂ℓ(xi,Θ)
∂Θ.Usingmini-batchesofexamples,asopposedtooneexampleatatime,ishelpfulinseveralways. First,thegradient
ofthelossoveramini-batchisanestimateofthegradient
overthetrainingset, whose qualityimprovesas thebatch
size increases. Second, computation over a batch can be
much more efﬁcient than mcomputations for individual
examples, due to the parallelism afforded by the modern
computingplatforms.
While stochastic gradient is simple and effective, it
requires careful tuning of the model hyper-parameters,
speciﬁcallythelearningrateusedinoptimization,aswell


requires careful tuning of the model hyper-parameters,
speciﬁcallythelearningrateusedinoptimization,aswell
as the initial values for the model parameters. The trainingiscomplicatedbythefactthattheinputstoeachlayer
areaffectedbytheparametersofallprecedinglayers–so
that small changes to the network parameters amplify as
thenetworkbecomesdeeper.
The change in the distributions of layers’ inputs
presents a problem because the layers need to continuously adapt to the new distribution. When the input distributiontoalearningsystemchanges,itissaidtoexperiencecovariateshift (Shimodaira, 2000). This is typically
handled via domain adaptation (Jiang, 2008). However,
the notion of covariate shift can be extended beyond the
learningsystemasawhole,toapplytoitsparts,suchasa
sub-networkora layer. Considera networkcomputing
ℓ=F2(F1(u,Θ1),Θ2)
whereF1andF2are arbitrary transformations, and the
parameters Θ1,Θ2are to be learned so as to minimize
the lossℓ. Learning Θ2can be viewed as if the inputs
x =F1(u,Θ1)arefedintothesub-network


the lossℓ. Learning Θ2can be viewed as if the inputs
x =F1(u,Θ1)arefedintothesub-network
ℓ=F2(x,Θ2).
Forexample,agradientdescentstep
Θ2←Θ2−α
mm/summationdisplay
i=1∂F2(xi,Θ2)
∂Θ2
(forbatchsize mandlearningrate α)isexactlyequivalent
to that for a stand-alone network F2with input x. Therefore, the input distribution properties that make training
more efﬁcient – such as having the same distribution between the training and test data – apply to training the
sub-network as well. As such it is advantageous for the
distributionof xtoremainﬁxedovertime. Then, Θ2does
1
not have to readjust to compensate for the change in the
distributionof x.
Fixed distribution of inputs to a sub-network would
havepositiveconsequencesforthelayers outsidethesubnetwork,as well. Consider a layer with a sigmoid activation function z =g(Wu+b)whereuis the layer input,
the weight matrix Wand bias vector bare the layer parameters to be learned, and g(x) =1


the weight matrix Wand bias vector bare the layer parameters to be learned, and g(x) =1
1+exp(−x). As|x|
increases, g′(x)tends to zero. This means that for all dimensionsof x =Wu+bexceptthosewithsmallabsolute
values,thegradientﬂowingdownto uwillvanishandthe
model will train slowly. However, since xis affected by
W,band the parameters of all the layers below, changes
tothoseparametersduringtrainingwilllikelymovemany
dimensions of xinto the saturated regime of the nonlinearity and slow down the convergence. This effect is
ampliﬁed as the network depth increases. In practice,
the saturation problem and the resulting vanishing gradientsareusuallyaddressedbyusingRectiﬁedLinearUnits
(Nair&Hinton, 2010) ReLU(x) = max( x,0), careful
initialization (Bengio&Glorot, 2010; Saxeet al., 2013),
and small learning rates. If, however, we could ensure
that the distribution of nonlinearity inputs remains more
stable as the network trains, then the optimizer would be
less likely to get stuck in the saturated regime, and the
trainingwouldaccelerate.
We refer to the change in the distributions of internal


less likely to get stuck in the saturated regime, and the
trainingwouldaccelerate.
We refer to the change in the distributions of internal
nodes of a deep network, in the course of training, as Internal Covariate Shift . Eliminating it offers a promise of
faster training. We propose a new mechanism, which we
callBatch Normalization , that takes a step towards reducing internal covariate shift, and in doing so dramatically accelerates the training of deep neural nets. It accomplishes this via a normalization step that ﬁxes the
meansandvariancesoflayerinputs. BatchNormalization
also has a beneﬁcial effect on the gradient ﬂow through
the network, by reducing the dependence of gradients
on the scale of the parameters or of their initial values.
This allows us to use much higher learning rates without the risk of divergence. Furthermore, batch normalization regularizes the model and reduces the need for
Dropout(Srivastavaet al., 2014). Finally, Batch Normalization makes it possible to use saturating nonlinearities
by preventingthe network from getting stuck in the saturatedmodes.
In Sec. 4.2, we apply Batch Normalization to the bestperforming ImageNet classiﬁcation network, and show
that we can match its performance using only 7% of the


that we can match its performance using only 7% of the
training steps, and can further exceed its accuracy by a
substantial margin. Using an ensemble of such networks
trained with Batch Normalization, we achieve the top-5
error rate that improves upon the best known results on
ImageNetclassiﬁcation.2 Towards Reducing Internal
CovariateShift
We deﬁne Internal Covariate Shift as the change in the
distribution of network activations due to the change in
networkparametersduringtraining. Toimprovethetraining, we seek to reduce the internal covariate shift. By
ﬁxingthe distributionof the layer inputs xas the training
progresses,weexpecttoimprovethetrainingspeed. Ithas
been long known (LeCunetal., 1998b; Wiesler &Ney,
2011) that the network training convergesfaster if its inputsarewhitened–i.e.,linearlytransformedtohavezero
meansandunitvariances,anddecorrelated. Aseachlayer
observestheinputsproducedbythelayersbelow,itwould
be advantageousto achieve the same whiteningof the inputsof each layer. By whitening the inputsto each layer,


be advantageousto achieve the same whiteningof the inputsof each layer. By whitening the inputsto each layer,
we would take a step towards achieving the ﬁxed distributions of inputs that would remove the ill effects of the
internalcovariateshift.
We couldconsiderwhiteningactivationsat everytraining step or at some interval, either by modifying the
network directly or by changing the parameters of the
optimization algorithm to depend on the network activation values (Wiesleret al., 2014; Raikoetal., 2012;
Poveyet al., 2014; Desjardins&Kavukcuoglu). However, if these modiﬁcations are interspersed with the optimization steps, then the gradient descent step may attempt to update the parameters in a way that requires
the normalization to be updated, which reduces the effect of the gradient step. For example, consider a layer
with the input uthat addsthe learned bias b, and normalizes the result by subtracting the mean of the activation
computed over the training data: /hatwidex=x−E[x]where
x=u+b,X={x1...N}is the set of values of xover
the training set, and E [x] =1
N/summationtextN
i=1xi. If a gradient


the training set, and E [x] =1
N/summationtextN
i=1xi. If a gradient
descent step ignores the dependence of E [x]onb, then it
will update b←b+ ∆b, where∆b∝−∂ℓ/∂/hatwidex. Then
u+ (b+ ∆b)−E[u+ (b+ ∆b)] =u+b−E[u+b].
Thus, the combination of the update to band subsequent
change in normalization led to no change in the output
of the layer nor, consequently, the loss. As the training
continues, bwill grow indeﬁnitely while the loss remains
ﬁxed. Thisproblemcangetworseifthenormalizationnot
only centers but also scales the activations. We have observed this empirically in initial experiments, where the
model blows up when the normalization parameters are
computedoutsidethe gradientdescentstep.
The issue with the above approach is that the gradient
descent optimization does not take into account the fact
that the normalization takes place. To address this issue,
we would like to ensure that, for any parameter values,
the network alwaysproducesactivationswith the desired
distribution. Doing so would allow the gradient of the


we would like to ensure that, for any parameter values,
the network alwaysproducesactivationswith the desired
distribution. Doing so would allow the gradient of the
loss with respect to the model parameters to account for
the normalization, and for its dependence on the model
parameters Θ. Let again xbe a layer input, treated as a
2
vector, andXbe the set of these inputs over the training
dataset. Thenormalizationcanthenbewrittenasatransformation
/hatwidex =Norm(x,X)
which depends not only on the given training example x
but on all examples X– each of which depends on Θif
xis generatedby anotherlayer. For backpropagation,we
wouldneedtocomputetheJacobians
∂Norm(x,X)
∂xand∂Norm(x,X)
∂X;
ignoring the latter term would lead to the explosion describedabove. Withinthisframework,whiteningthelayer
inputs is expensive, as it requires computing the covariance matrix Cov [x] =Ex∈X[xxT]−E[x]E[x]Tand its
inverse square root, to produce the whitened activations
Cov[x]−1/2(x−E[x]), as well as the derivatives of these


inverse square root, to produce the whitened activations
Cov[x]−1/2(x−E[x]), as well as the derivatives of these
transformsforbackpropagation.Thismotivatesustoseek
an alternative that performs input normalization in a way
that is differentiable and does not require the analysis of
theentiretrainingset aftereveryparameterupdate.
Some of the previous approaches (e.g.
(Lyu&Simoncelli, 2008)) use statistics computed
over a single training example, or, in the case of image
networks, over differentfeature maps at a given location.
However, this changes the representation ability of a
network by discarding the absolute scale of activations.
We want to a preservethe informationin the network,by
normalizing the activations in a training example relative
tothe statisticsoftheentiretrainingdata.
3 Normalization via Mini-Batch
Statistics
Since the full whitening of each layer’s inputs is costly
and not everywhere differentiable, we make two necessary simpliﬁcations. The ﬁrst is that instead of whitening
the features in layer inputs and outputs jointly, we will
normalizeeachscalarfeatureindependently,bymakingit
have the mean of zero and the variance of 1. For a layer


normalizeeachscalarfeatureindependently,bymakingit
have the mean of zero and the variance of 1. For a layer
withd-dimensionalinput x = (x(1)...x(d)),wewillnormalizeeachdimension
/hatwidex(k)=x(k)−E[x(k)]/radicalbig
Var[x(k)]
wheretheexpectationandvariancearecomputedoverthe
trainingdataset. Asshownin(LeCunetal.,1998b),such
normalizationspeedsupconvergence,evenwhenthefeaturesarenotdecorrelated.
Notethatsimplynormalizingeachinputofalayermay
change what the layer can represent. For instance, normalizing the inputsof a sigmoid wouldconstrain them to
the linear regime of the nonlinearity. To address this, we
makesurethat thetransformationinsertedin thenetwork
can represent the identity transform . To accomplish this,weintroduce,foreachactivation x(k),apairofparameters
γ(k),β(k),whichscale andshift thenormalizedvalue:
y(k)=γ(k)/hatwidex(k)+β(k).
These parameters are learned along with the original
model parameters, and restore the representation power


y(k)=γ(k)/hatwidex(k)+β(k).
These parameters are learned along with the original
model parameters, and restore the representation power
ofthenetwork. Indeed,bysetting γ(k)=/radicalbig
Var[x(k)]and
β(k)=E[x(k)], we couldrecoverthe originalactivations,
ifthatwerethe optimalthingto do.
Inthebatchsettingwhereeachtrainingstepisbasedon
theentire trainingset, we woulduse the wholeset to normalize activations. However,this is impracticalwhen using stochastic optimization. Therefore, we make the secondsimpliﬁcation: since we use mini-batchesin stochastic gradient training, each mini-batch produces estimates
ofthemeanandvariance ofeachactivation. Thisway,the
statistics used for normalization can fully participate in
the gradient backpropagation. Note that the use of minibatchesis enabledbycomputationof per-dimensionvariances rather than joint covariances; in the joint case, regularizationwouldbe requiredsince the mini-batchsize is
likely to be smaller than the number of activations being
whitened,resultinginsingularcovariancematrices.


likely to be smaller than the number of activations being
whitened,resultinginsingularcovariancematrices.
Consider a mini-batch Bof sizem. Since the normalization is applied to each activation independently, let us
focusonaparticularactivation x(k)andomitkforclarity.
We havemvaluesofthisactivationinthemini-batch,
B={x1...m}.
Letthenormalizedvaluesbe /hatwidex1...m,andtheirlineartransformationsbe y1...m. We referto thetransform
BNγ,β:x1...m→y1...m
as theBatch Normalizing Transform . We present the BN
TransforminAlgorithm1. Inthealgorithm, ǫisaconstant
addedtothemini-batchvariancefornumericalstability.
Input:Valuesof xovera mini-batch: B={x1...m};
Parametersto belearned: γ,β
Output:{yi=BNγ,β(xi)}
µB←1
mm/summationdisplay
i=1xi // mini-batchmean
σ2
B←1
mm/summationdisplay
i=1(xi−µB)2// mini-batchvariance


σ2
B←1
mm/summationdisplay
i=1(xi−µB)2// mini-batchvariance
/hatwidexi←xi−µB/radicalbig
σ2
B+ǫ// normalize
yi←γ/hatwidexi+β≡BNγ,β(xi) // scale andshift
Algorithm 1: Batch Normalizing Transform, applied to
activation xoveramini-batch.
TheBNtransformcanbeaddedtoanetworktomanipulate any activation. In the notation y=BNγ,β(x), we
3
indicate that the parameters γandβare to be learned,
but it should be noted that the BN transform does not
independently process the activation in each training example. Rather, BN γ,β(x)depends both on the training
exampleand the other examples in the mini-batch . The
scaled and shifted values yare passed to other network
layers. The normalized activations /hatwidexare internal to our
transformation, but their presence is crucial. The distributions of values of any /hatwidexhas the expected value of 0
and the variance of 1, as long as the elements of each
mini-batch are sampled from the same distribution, and


and the variance of 1, as long as the elements of each
mini-batch are sampled from the same distribution, and
if we neglect ǫ. This can be seen by observing that/summationtextm
i=1/hatwidexi= 0and1
m/summationtextm
i=1/hatwidex2
i= 1, and taking expectations. Eachnormalizedactivation /hatwidex(k)canbeviewedas
an input to a sub-network composed of the linear transformy(k)=γ(k)/hatwidex(k)+β(k), followed by the other processing doneby the originalnetwork. Thesesub-network
inputs all have ﬁxed means and variances, and although
the jointdistributionofthese normalized /hatwidex(k)canchange
over the course of training, we expect that the introduction of normalized inputs accelerates the training of the
sub-networkand,consequently,thenetworkasawhole.
During training we need to backpropagate the gradient of loss ℓthrough this transformation,as well as compute the gradients with respect to the parameters of the
BN transform. We use chainrule,as follows(beforesimpliﬁcation):
∂ℓ
∂/hatwidexi=∂ℓ
∂yi·γ


∂ℓ
∂/hatwidexi=∂ℓ
∂yi·γ
∂ℓ
∂σ2
B=/summationtextm
i=1∂ℓ
∂/hatwidexi·(xi−µB)·−1
2(σ2
B+ǫ)−3/2
∂ℓ
∂µB=/parenleftbigg/summationtextm
i=1∂ℓ
∂/hatwidexi·−1√
σ2
B+ǫ/parenrightbigg
+∂ℓ
∂σ2
B·/summationtextm
i=1−2(xi−µB)
m
∂ℓ
∂xi=∂ℓ
∂/hatwidexi·1√
σ2
B+ǫ+∂ℓ
∂σ2
B·2(xi−µB)
m+∂ℓ
∂µB·1
m
∂ℓ
∂γ=/summationtextm
i=1∂ℓ
∂yi·/hatwidexi
∂ℓ


∂ℓ
∂γ=/summationtextm
i=1∂ℓ
∂yi·/hatwidexi
∂ℓ
∂β=/summationtextm
i=1∂ℓ
∂yi
Thus,BNtransformisadifferentiabletransformationthat
introduces normalized activations into the network. This
ensures that as the model is training, layers can continue
learningoninputdistributionsthatexhibitlessinternal covariate shift, thus accelerating the training. Furthermor e,
the learned afﬁne transform applied to these normalized
activationsallowstheBNtransformtorepresenttheidentity transformationandpreservesthenetworkcapacity.
3.1 Training and Inference with BatchNormalizedNetworks
ToBatch-Normalize anetwork,wespecifyasubsetofactivations and insert the BN transform for each of them,
according to Alg. 1. Any layer that previously received
xas the input, now receives BN (x). A model employing
Batch Normalization can be trained using batch gradient
descent,orStochasticGradientDescentwithamini-batch
sizem >1, or with any of its variants such as Adagrad(Duchiet al.,2011). Thenormalizationofactivationsthat


sizem >1, or with any of its variants such as Adagrad(Duchiet al.,2011). Thenormalizationofactivationsthat
dependsonthemini-batchallowsefﬁcienttraining,but is
neithernecessarynordesirableduringinference;wewant
the output to depend only on the input, deterministically.
For this, once the network has been trained, we use the
normalization
/hatwidex=x−E[x]/radicalbig
Var[x]+ǫ
using the population, rather than mini-batch, statistics.
Neglecting ǫ, these normalized activations have the same
mean0 and variance1 as duringtraining. We use the unbiased variance estimate Var [x] =m
m−1·EB[σ2
B], where
theexpectationisovertrainingmini-batchesofsize mand
σ2
Baretheirsamplevariances. Usingmovingaveragesinstead, we can track the accuracy of a model as it trains.
Sincethemeansandvariancesareﬁxedduringinference,
the normalization is simply a linear transform applied to
eachactivation. Itmayfurtherbecomposedwiththescaling byγand shift by β, to yield a single linear transform


the normalization is simply a linear transform applied to
eachactivation. Itmayfurtherbecomposedwiththescaling byγand shift by β, to yield a single linear transform
that replacesBN (x). Algorithm 2 summarizesthe procedurefortrainingbatch-normalizednetworks.
Input:NetworkNwith trainableparameters Θ;
subsetofactivations {x(k)}K
k=1
Output: Batch-normalizednetworkforinference, Ninf
BN
1:Ntr
BN←N// TrainingBN network
2:fork= 1...Kdo
3:Add transformation y(k)=BNγ(k),β(k)(x(k))to
Ntr
BN(Alg.1)
4:Modify each layer in Ntr
BNwith input x(k)to take
y(k)instead
5:end for
6:TrainNtr
BNto optimize the parameters Θ∪
{γ(k),β(k)}K
k=1
7:Ninf
BN←Ntr
BN// InferenceBN networkwithfrozen
// parameters
8:fork= 1...Kdo
9:// Forclarity, x≡x(k),γ≡γ(k),µB≡µ(k)
B, etc.


9:// Forclarity, x≡x(k),γ≡γ(k),µB≡µ(k)
B, etc.
10:Process multiple training mini-batches B, each of
sizem,andaverageoverthem:
E[x]←EB[µB]
Var[x]←m
m−1EB[σ2
B]
11:InNinf
BN, replace the transform y=BNγ,β(x)with
y=γ√
Var[x]+ǫ·x+/parenleftbig
β−γE[x]√
Var[x]+ǫ/parenrightbig
12:end for
Algorithm2: Traininga Batch-NormalizedNetwork
3.2 Batch-Normalized Convolutional Networks
Batch Normalization can be applied to any set of activations in the network. Here, we focus on transforms
4
that consist of an afﬁne transformation followed by an
element-wisenonlinearity:
z =g(Wu+b)
whereWandbare learned parametersof the model, and
g(·)isthenonlinearitysuchassigmoidorReLU.Thisformulation covers both fully-connected and convolutional
layers. We add the BN transform immediately before the


layers. We add the BN transform immediately before the
nonlinearity,bynormalizing x =Wu+b. Wecouldhave
also normalized the layer inputs u, but since uis likely
the output of another nonlinearity, the shape of its distributionislikelytochangeduringtraining,andconstrainin g
its ﬁrst and second moments would not eliminate the covariate shift. In contrast, Wu + bis more likely to have
a symmetric,non-sparsedistribution,that is “moreGaussian”(Hyv¨ arinen&Oja,2000);normalizingitislikelyto
produceactivationswithastable distribution.
Notethat,sincewenormalize Wu+b,thebiasbcanbe
ignoredsinceitseffectwillbecanceledbythesubsequent
meansubtraction(theroleofthebiasissubsumedby βin
Alg.1). Thus, z =g(Wu+b)is replacedwith
z =g(BN(Wu))
where the BN transformis applied independentlyto each
dimension of x =Wu, with a separate pair of learned
parameters γ(k),β(k)perdimension.
Forconvolutionallayers,we additionallywant the normalization to obey the convolutional property – so that


parameters γ(k),β(k)perdimension.
Forconvolutionallayers,we additionallywant the normalization to obey the convolutional property – so that
different elements of the same feature map, at different
locations, are normalized in the same way. To achieve
this, we jointly normalize all the activations in a minibatch, overall locations. In Alg. 1, we let Bbe the set of
all values in a feature map across both the elements of a
mini-batch and spatial locations – so for a mini-batch of
sizemand feature maps of size p×q, we use the effective mini-batch of size m′=|B|=m·pq. We learn a
pair of parameters γ(k)andβ(k)per feature map, rather
than per activation. Alg. 2 is modiﬁed similarly, so that
duringinferencetheBNtransformappliesthesamelinear
transformationtoeachactivationina givenfeaturemap.
3.3 Batch Normalization enables higher
learning rates
In traditional deep networks, too-high learning rate may
result in the gradients that explode or vanish, as well as
getting stuck in poor local minima. Batch Normalization helps address these issues. By normalizing activations throughout the network, it prevents small changes


getting stuck in poor local minima. Batch Normalization helps address these issues. By normalizing activations throughout the network, it prevents small changes
to the parameters from amplifying into larger and suboptimal changes in activations in gradients; for instance, it
prevents the training from getting stuck in the saturated
regimesofnonlinearities.
BatchNormalizationalsomakestrainingmoreresilient
totheparameterscale. Normally,largelearningratesmay
increasethescaleoflayerparameters,whichthenamplifythegradientduringbackpropagationandleadtothemodel
explosion. However, with Batch Normalization, backpropagation through a layer is unaffected by the scale of
itsparameters. Indeed,fora scalar a,
BN(Wu) =BN((aW)u)
andwe canshowthat
∂BN((aW)u)
∂u=∂BN(Wu)
∂u
∂BN((aW)u)
∂(aW)=1
a·∂BN(Wu)
∂W
The scale does not affect the layer Jacobian nor, consequently, the gradient propagation. Moreover, larger
weights lead to smallergradients, and Batch Normalizationwill stabilize theparametergrowth.
We further conjecture that Batch Normalization may