References
[1] Pablo Arbelaez, Jordi Pont-Tuset, Jon Barron, Ferran Marques, and Jitendra Malik.
Multiscale combinatorial grouping. In CVPR , pages 328–335. IEEE, 2014.
[2] Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, and Philip H. S. Torr. Higher order
conditional random ﬁelds in deep neural networks. In ECCV , 2016.
[3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L
Yuille. Semantic image segmentation with deep convolutional nets and fully connected
crfs. ICLR , 2015.
[4] Yi-Ting Chen, Xiaokai Liu, and Ming-Hsuan Yang. Multi-instance object segmentation
with occlusion handling. In CVPR , pages 3470–3478, 2015.
[5] Jifeng Dai, Kaiming He, and Jian Sun. Convolutional feature masking for joint object
and stuff segmentation. In CVPR , 2015.
[6] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise
convolutional networks for semantic segmentation. In ICCV , 2015.


convolutional networks for semantic segmentation. In ICCV , 2015.
ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 11
[7] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via
multi-task network cascades. In CVPR , 2016.
[8] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew
Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010.
[9] Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively
trained, multiscale, deformable part model. In CVPR , pages 1–8. IEEE, 2008.
[10] Spyros Gidaris and Nikos Komodakis. Object detection via a multi-region and semantic
segmentation-aware cnn model. In ICCV , pages 1134–1142, 2015.
[11] Ross Girshick. Fast r-cnn. In ICCV , 2015.
[12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , 2014.

Bottom-up Instance Segmentation using Deep Higher-Order CRFs

Summary:
Traditional Scene Understanding problems such as Object Detection and
Semantic Segmentation have made breakthroughs in recent years due to the
adoption of deep learning. However, the former task is not able to localise
objects at a pixel level, and the latter task has no notion of different
instances of objects of the same class. We focus on the task of Instance
Segmentation which recognises and localises objects down to a pixel level. Our
model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable
higher order potentials based on object detector outputs. This allows us to
reason about instances from an initial, category-level semantic segmentation.
Our simple method effectively leverages the great progress recently made in
semantic segmentation and object detection. The accurate instance-level
segmentations that our network produces is reflected by the considerable
improvements obtained over previous work.

British Machine Vision Conference (BMVC) 2016


[13] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous
detection and segmentation. In ECCV , pages 297–312. Springer, 2014.
[14] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns
for object segmentation and ﬁne-grained localization. CVPR , 2015.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in
deep convolutional networks for visual recognition. In ECCV , 2014.
[16] Xuming He, Richard S Zemel, and Miguel Á Carreira-Perpiñán. Multiscale conditional
random ﬁelds for image labeling. In CVPR , volume 2, pages II–695. IEEE, 2004.
[17] Pushmeet Kohli, Lubor Ladicky, and Philip H.S. Torr. Robust higher order potentials
for enforcing label consistency. IJCV , 82(3):302–324, 2009.
[18] P. Krähenbühl and V . Koltun. Efﬁcient inference in fully connected CRFs with Gaussian edge potentials. In NIPS , 2011.


[19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classiﬁcation with
deep convolutional neural networks. In NIPS , pages 1097–1105. 2012.
[20] Lubor Ladicky, Chris Russell, Pushmeet Kohli, and Philip HS Torr. Graph cut based
inference with co-occurrence statistics. In ECCV , pages 239–253. 2010.
[21] L’ubor Ladický, Paul Sturgess, Karteek Alahari, Chris Russell, and Philip H. S. Torr.
What, where and how many? combining object detectors and crfs. In ECCV , pages
424–437, 2010.
[22] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324,
1998.
[23] Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Zequn Jie, Jiashi Feng, Liang Lin, and
Shuicheng Yan. Reversible recursive instance-level object segmentation. arXiv preprint
arXiv:1511.04517 , 2015.


Shuicheng Yan. Reversible recursive instance-level object segmentation. arXiv preprint
arXiv:1511.04517 , 2015.
12 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
[24] Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Jianchao Yang, Liang Lin, and Shuicheng
Yan. Proposal-free network for instance-level object segmentation. arXiv preprint
arXiv:1509.02636 , 2015.
[25] Guosheng Lin, Chunhua Shen, Ian Reid, et al. Efﬁcient piecewise training of deep
structured models for semantic segmentation. In CVPR , 2016.
[26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in
context. In ECCV , pages 740–755. Springer, 2014.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for
semantic segmentation. In CVPR , 2015.
[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards


[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards
real-time object detection with region proposal networks. In NIPS , 2015.
[29] Bernardino Romera-Paredes and Philip HS Torr. Recurrent instance segmentation. In
ECCV , 2016.
[30] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM TOG , 2004.
[31] Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost for
image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. IJCV , 2009.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largescale image recognition. In ICLR , 2015.
[33] Min Sun, Byung-soo Kim, Pushmeet Kohli, and Silvio Savarese. Relating things and
stuff via object property interactions. PAMI , 36(7):1370–1383, 2014.
[34] Jasper RR Uijlings, Koen EA van de Sande, Theo Gevers, and Arnold WM Smeulders.


[34] Jasper RR Uijlings, Koen EA van de Sande, Theo Gevers, and Arnold WM Smeulders.
Selective search for object recognition. IJCV , 104(2):154–171, 2013.
[35] Vibhav Vineet, Jonathan Warrell, and Philip HS Torr. Filter-based mean-ﬁeld inference
for random ﬁelds with higher-order terms and product label-spaces. IJCV , 2014.
[36] Jian Yao, Sanja Fidler, and Raquel Urtasun. Describing the Scene as a Whole: Joint
Object Detection, Scene Classiﬁcation and Semantic Segmentation. In CVPR , pages
702–709, 2012.
[37] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet,
Zhizhong Su, Dalong Du, Chang Huang, and Philip Torr. Conditional random ﬁelds as
recurrent neural networks. In ICCV , 2015.


Aggregated Residual Transformations for Deep Neural Networks
Saining Xie1Ross Girshick2Piotr Doll ´ar2Zhuowen Tu1Kaiming He2
1UC San Diego2Facebook AI Research
fs9xie,ztug@ucsd.edu frbg,pdollar,kaiminghe g@fb.com
Abstract
We present a simple, highly modularized network architecture for image classiﬁcation. Our network is constructed
by repeating a building block that aggregates a set of transformations with the same topology. Our simple design results in a homogeneous, multi-branch architecture that has
only a few hyper-parameters to set. This strategy exposes a
new dimension, which we call “cardinality” (the size of the
set of transformations), as an essential factor in addition to
the dimensions of depth and width. On the ImageNet-1K
dataset, we empirically show that even under the restricted
condition of maintaining complexity, increasing cardinality
is able to improve classiﬁcation accuracy. Moreover, increasing cardinality is more effective than going deeper or
wider when we increase the capacity. Our models, named
ResNeXt, are the foundations of our entry to the ILSVRC
2016 classiﬁcation task in which we secured 2nd place.

Summary:
We present a simple, highly modularized network architecture for image
classification. Our network is constructed by repeating a building block that
aggregates a set of transformations with the same topology. Our simple design
results in a homogeneous, multi-branch architecture that has only a few
hyper-parameters to set. This strategy exposes a new dimension, which we call
"cardinality" (the size of the set of transformations), as an essential factor
in addition to the dimensions of depth and width. On the ImageNet-1K dataset,
we empirically show that even under the restricted condition of maintaining
complexity, increasing cardinality is able to improve classification accuracy.
Moreover, increasing cardinality is more effective than going deeper or wider
when we increase the capacity. Our models, named ResNeXt, are the foundations
of our entry to the ILSVRC 2016 classification task in which we secured 2nd
place. We further investigate ResNeXt on an ImageNet-5K set and the COCO
detection set, also showing better results than its ResNet counterpart. The
code and models are publicly available online.

Saining Xie
Ross Girshick
Piotr Dollár
Zhuowen Tu
Kaiming He

Accepted to CVPR 2017. Code and models:
  https://github.com/facebookresearch/ResNeXt


ResNeXt, are the foundations of our entry to the ILSVRC
2016 classiﬁcation task in which we secured 2nd place.
We further investigate ResNeXt on an ImageNet-5K set and
the COCO detection set, also showing better results than
its ResNet counterpart. The code and models are publicly
available online1.
1. Introduction
Research on visual recognition is undergoing a transition from “feature engineering” to “network engineering”
[25, 24, 44, 34, 36, 38, 14]. In contrast to traditional handdesigned features ( e.g., SIFT [29] and HOG [5]), features
learned by neural networks from large-scale data [33] require minimal human involvement during training, and can
be transferred to a variety of recognition tasks [7, 10, 28].
Nevertheless, human effort has been shifted to designing
better network architectures for learning representations.
Designing architectures becomes increasingly difﬁcult
with the growing number of hyper-parameters (width2, ﬁlter sizes, strides, etc.), especially when there are many layers. The VGG-nets [36] exhibit a simple yet effective strategy of constructing very deep networks: stacking build1https://github.com/facebookresearch/ResNeXt
2Width refers to the number of channels in a layer.


2Width refers to the number of channels in a layer.
256, 1x1, 4
4, 3x3, 4
4, 1x1, 256
+256, 1x1, 4
4, 3x3, 4
4, 1x1, 256256, 1x1, 4
4, 3x3, 4
4, 1x1, 256....total 32
paths256-d in
+256, 1x1, 64
64, 3x3, 64
64, 1x1, 256
+256-d in
256-d out
256-d outFigure 1. Left: A block of ResNet [14]. Right : A block of
ResNeXt with cardinality = 32 , with roughly the same complexity. A layer is shown as (# in channels, ﬁlter size, # out channels).
ing blocks of the same shape. This strategy is inherited
by ResNets [14] which stack modules of the same topology. This simple rule reduces the free choices of hyperparameters, and depth is exposed as an essential dimension
in neural networks. Moreover, we argue that the simplicity


in neural networks. Moreover, we argue that the simplicity
of this rule may reduce the risk of over-adapting the hyperparameters to a speciﬁc dataset. The robustness of VGGnets and ResNets has been proven by various visual recognition tasks [7, 10, 9, 28, 31, 14] and by non-visual tasks
involving speech [42, 30] and language [4, 41, 20].
Unlike VGG-nets, the family of Inception models [38,
17, 39, 37] have demonstrated that carefully designed
topologies are able to achieve compelling accuracy with low
theoretical complexity. The Inception models have evolved
over time [38, 39], but an important common property is
asplit-transform-merge strategy. In an Inception module,
the input is split into a few lower-dimensional embeddings
(by 11 convolutions), transformed by a set of specialized
ﬁlters (33, 55,etc.), and merged by concatenation. It
can be shown that the solution space of this architecture is a
strict subspace of the solution space of a single large layer
(e.g., 55) operating on a high-dimensional embedding.
The split-transform-merge behavior of Inception modules


(e.g., 55) operating on a high-dimensional embedding.
The split-transform-merge behavior of Inception modules
is expected to approach the representational power of large
and dense layers, but at a considerably lower computational
complexity.
Despite good accuracy, the realization of Inception models has been accompanied with a series of complicating fac1arXiv:1611.05431v2  [cs.CV]  11 Apr 2017
tors — the ﬁlter numbers and sizes are tailored for each
individual transformation, and the modules are customized
stage-by-stage. Although careful combinations of these
components yield excellent neural network recipes, it is in
general unclear how to adapt the Inception architectures to
new datasets/tasks, especially when there are many factors
and hyper-parameters to be designed.
In this paper, we present a simple architecture which
adopts VGG/ResNets’ strategy of repeating layers, while
exploiting the split-transform-merge strategy in an easy, extensible way. A module in our network performs a set
of transformations, each on a low-dimensional embedding,
whose outputs are aggregated by summation. We pursuit a
simple realization of this idea — the transformations to be
aggregated are all of the same topology ( e.g., Fig. 1 (right)).


simple realization of this idea — the transformations to be
aggregated are all of the same topology ( e.g., Fig. 1 (right)).
This design allows us to extend to any large number of
transformations without specialized designs.
Interestingly, under this simpliﬁed situation we show that
our model has two other equivalent forms (Fig. 3). The reformulation in Fig. 3(b) appears similar to the InceptionResNet module [37] in that it concatenates multiple paths;
but our module differs from all existing Inception modules
in that all our paths share the same topology and thus the
number of paths can be easily isolated as a factor to be investigated. In a more succinct reformulation, our module
can be reshaped by Krizhevsky et al .’s grouped convolutions [24] (Fig. 3(c)), which, however, had been developed
as an engineering compromise.
We empirically demonstrate that our aggregated transformations outperform the original ResNet module, even
under the restricted condition of maintaining computational
complexity and model size — e.g., Fig. 1(right) is designed
to keep the FLOPs complexity and number of parameters of
Fig. 1(left). We emphasize that while it is relatively easy to
increase accuracy by increasing capacity (going deeper or
wider), methods that increase accuracy while maintaining


Fig. 1(left). We emphasize that while it is relatively easy to
increase accuracy by increasing capacity (going deeper or
wider), methods that increase accuracy while maintaining
(or reducing) complexity are rare in the literature.
Our method indicates that cardinality (the size of the
set of transformations) is a concrete, measurable dimension that is of central importance, in addition to the dimensions of width and depth. Experiments demonstrate that increasing cardinality is a more effective way of gaining accuracy than going deeper or wider , especially when depth and
width starts to give diminishing returns for existing models.
Our neural networks, named ResNeXt (suggesting the
nextdimension), outperform ResNet-101/152 [14], ResNet200 [15], Inception-v3 [39], and Inception-ResNet-v2 [37]
on the ImageNet classiﬁcation dataset. In particular, a
101-layer ResNeXt is able to achieve better accuracy than
ResNet-200 [15] but has only 50% complexity. Moreover,
ResNeXt exhibits considerably simpler designs than all Inception models. ResNeXt was the foundation of our submission to the ILSVRC 2016 classiﬁcation task, in whichwe secured second place. This paper further evaluates
ResNeXt on a larger ImageNet-5K set and the COCO object


ResNeXt on a larger ImageNet-5K set and the COCO object
detection dataset [27], showing consistently better accuracy
than its ResNet counterparts. We expect that ResNeXt will
also generalize well to other visual (and non-visual) recognition tasks.
2. Related Work
Multi-branch convolutional networks . The Inception
models [38, 17, 39, 37] are successful multi-branch architectures where each branch is carefully customized.
ResNets [14] can be thought of as two-branch networks
where one branch is the identity mapping. Deep neural decision forests [22] are tree-patterned multi-branch networks
with learned splitting functions.
Grouped convolutions . The use of grouped convolutions
dates back to the AlexNet paper [24], if not earlier. The
motivation given by Krizhevsky et al. [24] is for distributing
the model over two GPUs. Grouped convolutions are supported by Caffe [19], Torch [3], and other libraries, mainly
for compatibility of AlexNet. To the best of our knowledge,
there has been little evidence on exploiting grouped convolutions to improve accuracy. A special case of grouped convolutions is channel-wise convolutions in which the number
of groups is equal to the number of channels. Channel-wise


of groups is equal to the number of channels. Channel-wise
convolutions are part of the separable convolutions in [35].
Compressing convolutional networks . Decomposition (at
spatial [6, 18] and/or channel [6, 21, 16] level) is a widely
adopted technique to reduce redundancy of deep convolutional networks and accelerate/compress them. Ioannouet al . [16] present a “root”-patterned network for reducing computation, and branches in the root are realized
by grouped convolutions. These methods [6, 18, 21, 16]
have shown elegant compromise of accuracy with lower
complexity and smaller model sizes. Instead of compression, our method is an architecture that empirically shows
stronger representational power.
Ensembling . Averaging a set of independently trained networks is an effective solution to improving accuracy [24],
widely adopted in recognition competitions [33]. Veit et al.
[40] interpret a single ResNet as an ensemble of shallower
networks, which results from ResNet’s additive behaviors
[15]. Our method harnesses additions to aggregate a set of
transformations. But we argue that it is imprecise to view
our method as ensembling, because the members to be aggregated are trained jointly, not independently.
3. Method
3.1. Template


our method as ensembling, because the members to be aggregated are trained jointly, not independently.
3. Method
3.1. Template
We adopt a highly modularized design following
VGG/ResNets. Our network consists of a stack of residstage output ResNet-50 ResNeXt-50 (32 4d)
conv1 112112 77, 64, stride 2 77, 64, stride 2
conv2 565633 max pool, stride 2 33 max pool, stride 22
6411, 64
33, 64
11, 2563
7532
6411, 128
33, 128, C=32
11, 2563
753
conv3 28282
6411, 128
33, 128
11, 5123
7542
6411, 256
33, 256, C=32
11, 5123
754
conv4 14142
6411, 256
33, 256
11, 10243
7562
6411, 512
33, 512, C=32
11, 10243
756
conv5 772
6411, 512
33, 512


33, 512, C=32
11, 10243
756
conv5 772
6411, 512
33, 512
11, 20483
7532
6411, 1024
33, 1024, C=32
11, 20483
753
11global average pool global average pool
1000-d fc, softmax 1000-d fc, softmax
# params. 25.510625.0106
FLOPs 4.11094.2109
Table 1. ( Left) ResNet-50. ( Right ) ResNeXt-50 with a 32 4d
template (using the reformulation in Fig. 3(c)). Inside the brackets
are the shape of a residual block, and outside the brackets is the
number of stacked blocks on a stage. “ C=32” suggests grouped
convolutions [24] with 32 groups. The numbers of parameters and
FLOPs are similar between these two models.
ual blocks. These blocks have the same topology, and are
subject to two simple rules inspired by VGG/ResNets: (i)
if producing spatial maps of the same size, the blocks share


subject to two simple rules inspired by VGG/ResNets: (i)
if producing spatial maps of the same size, the blocks share
the same hyper-parameters (width and ﬁlter sizes), and (ii)
each time when the spatial map is downsampled by a factor of 2, the width of the blocks is multiplied by a factor
of 2. The second rule ensures that the computational complexity, in terms of FLOPs (ﬂoating-point operations, in #
of multiply-adds), is roughly the same for all blocks.
With these two rules, we only need to design a template
module, and all modules in a network can be determined
accordingly. So these two rules greatly narrow down the
design space and allow us to focus on a few key factors.
The networks constructed by these rules are in Table 1.
3.2. Revisiting Simple Neurons
The simplest neurons in artiﬁcial neural networks perform inner product (weighted sum), which is the elementary transformation done by fully-connected and convolutional layers. Inner product can be thought of as a form of
aggregating transformation:
DX
i=1wixi; (1)
where x= [x1; x2; :::; x D]is aD-channel input vector to


DX
i=1wixi; (1)
where x= [x1; x2; :::; x D]is aD-channel input vector to
the neuron and wiis a ﬁlter’s weight for the i-th chan.......
+x1 x2 xD x3
w1w2w3wDxFigure 2. A simple neuron that performs inner product.
nel. This operation (usually including some output nonlinearity) is referred to as a “neuron”. See Fig. 2.
The above operation can be recast as a combination of
splitting, transforming, and aggregating . (i) Splitting : the
vector xis sliced as a low-dimensional embedding, and
in the above, it is a single-dimension subspace xi. (ii)
Transforming : the low-dimensional representation is transformed, and in the above, it is simply scaled: wixi. (iii)
Aggregating : the transformations in all embeddings are aggregated byPD
i=1.
3.3. Aggregated Transformations
Given the above analysis of a simple neuron, we consider replacing the elementary transformation ( wixi) with
a more generic function, which in itself can also be a network. In contrast to “Network-in-Network” [26] that turns


a more generic function, which in itself can also be a network. In contrast to “Network-in-Network” [26] that turns
out to increase the dimension of depth, we show that our
“Network-in- Neuron ” expands along a new dimension.
Formally, we present aggregated transformations as:
F(x) =CX
i=1Ti(x); (2)
whereTi(x)can be an arbitrary function. Analogous to a
simple neuron,Tishould project xinto an (optionally lowdimensional) embedding and then transform it.
In Eqn.(2), Cis the size of the set of transformations
to be aggregated. We refer to Cascardinality [2]. In
Eqn.(2) Cis in a position similar to Din Eqn.(1), but C
need not equal Dand can be an arbitrary number. While
the dimension of width is related to the number of simple
transformations (inner product), we argue that the dimension of cardinality controls the number of more complex
transformations. We show by experiments that cardinality
is an essential dimension and can be more effective than the
dimensions of width and depth.
In this paper, we consider a simple way of designing the
transformation functions: all Ti’s have the same topology.


dimensions of width and depth.
In this paper, we consider a simple way of designing the
transformation functions: all Ti’s have the same topology.
This extends the VGG-style strategy of repeating layers of
the same shape, which is helpful for isolating a few factors
and extending to any large number of transformations. We
set the individual transformation Tito be the bottleneckshaped architecture [14], as illustrated in Fig. 1 (right). In
this case, the ﬁrst 1 1 layer in eachTiproduces the lowdimensional embedding.
equivalent
256, 1x1, 4
4, 3x3, 4
4, 1x1, 256
+256, 1x1, 4
4, 3x3, 4
4, 1x1, 256256, 1x1, 4
4, 3x3, 4
4, 1x1, 256
(a)....total 32
paths256-d in
+256, 1x1, 4
4, 3x3, 4256, 1x1, 4
4, 3x3, 4256, 1x1, 4
4, 3x3, 4
+concatenate
128, 1x1, 256256-d in
....total 32
paths
(b)256, 1x1, 128


+concatenate
128, 1x1, 256256-d in
....total 32
paths
(b)256, 1x1, 128
128, 3x3, 128
group = 32
128, 1x1, 256
+256-d in
(c)256-d out 256-d out 256-d outFigure 3. Equivalent building blocks of ResNeXt. (a): Aggregated residual transformations, the same as Fig. 1 right. (b): A block equivalent
to (a), implemented as early concatenation. (c): A block equivalent to (a,b), implemented as grouped convolutions [24]. Notations in bold
text highlight the reformulation changes. A layer is denoted as (# input channels, ﬁlter size, # output channels).
The aggregated transformation in Eqn.(2) serves as the
residual function [14] (Fig. 1 right):
y=x+CX
i=1Ti(x); (3)
where yis the output.
Relation to Inception-ResNet . Some tensor manipulations show that the module in Fig. 1(right) (also shown in


where yis the output.
Relation to Inception-ResNet . Some tensor manipulations show that the module in Fig. 1(right) (also shown in
Fig. 3(a)) is equivalent to Fig. 3(b).3Fig. 3(b) appears similar to the Inception-ResNet [37] block in that it involves
branching and concatenating in the residual function. But
unlike all Inception or Inception-ResNet modules, we share
the same topology among the multiple paths. Our module
requires minimal extra effort designing each path.
Relation to Grouped Convolutions . The above module becomes more succinct using the notation of grouped convolutions [24].4This reformulation is illustrated in Fig. 3(c).
All the low-dimensional embeddings (the ﬁrst 1 1 layers)
can be replaced by a single, wider layer ( e.g., 11, 128-d
in Fig 3(c)). Splitting is essentially done by the grouped
convolutional layer when it divides its input channels into
groups. The grouped convolutional layer in Fig. 3(c) performs 32 groups of convolutions whose input and output
channels are 4-dimensional. The grouped convolutional
layer concatenates them as the outputs of the layer. The


channels are 4-dimensional. The grouped convolutional
layer concatenates them as the outputs of the layer. The
block in Fig. 3(c) looks like the original bottleneck residual block in Fig. 1(left), except that Fig. 3(c) is a wider but
sparsely connected module.
3An informal but descriptive proof is as follows. Note the equality:
A1B1+A2B2= [A1; A2][B1;B2]where [;]is horizontal concatenation and [ ; ]is vertical concatenation. Let Aibe the weight of the last layer
andBibe the output response of the second-last layer in the block. In the
case of C= 2, the element-wise addition in Fig. 3(a) is A1B1+A2B2,
the weight of the last layer in Fig. 3(b) is [A1; A2], and the concatenation
of outputs of second-last layers in Fig. 3(b) is [B1;B2].
4In a group conv layer [24], input and output channels are divided into
Cgroups, and convolutions are separately performed within each group.
equivalent
64, 3x3, 4 64, 3x3, 4 64, 3x3, 4
....total 32
paths


equivalent
64, 3x3, 4 64, 3x3, 4 64, 3x3, 4
....total 32
paths
+4, 3x3, 64
+4, 3x3, 64 4, 3x3, 6464, 3x3, 128
128, 3x3, 64
+Figure 4. ( Left): Aggregating transformations of depth = 2 .
(Right ): An equivalent block, which is trivially wider.
We note that the reformulations produce nontrivial
topologies only when the block has depth 3. If the block
has depth =2 (e.g., the basic block in [14]), the reformulations lead to trivially a wide, dense module. See the illustration in Fig. 4.
Discussion . We note that although we present reformulations that exhibit concatenation (Fig. 3(b)) or grouped convolutions (Fig. 3(c)), such reformulations are not always applicable for the general form of Eqn.(3), e.g., if the transformationTitakes arbitrary forms and are heterogenous.
We choose to use homogenous forms in this paper because
they are simpler and extensible. Under this simpliﬁed case,
grouped convolutions in the form of Fig. 3(c) are helpful for
easing implementation.
3.4. Model Capacity


grouped convolutions in the form of Fig. 3(c) are helpful for
easing implementation.
3.4. Model Capacity
Our experiments in the next section will show that
our models improve accuracy when maintaining the model
complexity and number of parameters. This is not only interesting in practice, but more importantly, the complexity
and number of parameters represent inherent capacity of
models and thus are often investigated as fundamental properties of deep networks [8].
When we evaluate different cardinalities Cwhile preserving complexity, we want to minimize the modiﬁcation
of other hyper-parameters. We choose to adjust the width of
cardinality C 1 2 4 8 32
width of bottleneck d 64 40 24 14 4
width of group conv. 64 80 96 112 128
Table 2. Relations between cardinality and width (for the template
of conv2), with roughly preserved complexity on a residual block.
The number of parameters is 70k for the template of conv2. The
number of FLOPs is 0.22 billion (# params 5656 for conv2).
the bottleneck ( e.g., 4-d in Fig 1(right)), because it can be
isolated from the input and output of the block. This strategy introduces no change to other hyper-parameters (depth
or input/output width of blocks), so is helpful for us to focus


isolated from the input and output of the block. This strategy introduces no change to other hyper-parameters (depth
or input/output width of blocks), so is helpful for us to focus
on the impact of cardinality.
In Fig. 1(left), the original ResNet bottleneck block [14]
has25664 + 336464 + 6425670k parameters and
proportional FLOPs (on the same feature map size). With
bottleneck width d, our template in Fig. 1(right) has:
C(256d+ 33dd+d256) (4)
parameters and proportional FLOPs. When C= 32 and
d= 4, Eqn.(4)70k. Table 2 shows the relationship between cardinality Cand bottleneck width d.
Because we adopt the two rules in Sec. 3.1, the above
approximate equality is valid between a ResNet bottleneck
block and our ResNeXt on all stages (except for the subsampling layers where the feature maps size changes). Table 1 compares the original ResNet-50 and our ResNeXt-50
that is of similar capacity.5We note that the complexity can
only be preserved approximately, but the difference of the
complexity is minor and does not bias our results.
4. Implementation details


only be preserved approximately, but the difference of the
complexity is minor and does not bias our results.
4. Implementation details
Our implementation follows [14] and the publicly available code of fb.resnet.torch [11]. On the ImageNet
dataset, the input image is 224 224 randomly cropped
from a resized image using the scale and aspect ratio augmentation of [38] implemented by [11]. The shortcuts are
identity connections except for those increasing dimensions
which are projections (type B in [14]). Downsampling of
conv3, 4, and 5 is done by stride-2 convolutions in the 3 3
layer of the ﬁrst block in each stage, as suggested in [11].
We use SGD with a mini-batch size of 256 on 8 GPUs (32
per GPU). The weight decay is 0.0001 and the momentum
is 0.9. We start from a learning rate of 0.1, and divide it by
10 for three times using the schedule in [11]. We adopt the
weight initialization of [13]. In all ablation comparisons, we
evaluate the error on the single 224 224 center crop from
an image whose shorter side is 256.
Our models are realized by the form of Fig. 3(c). We


evaluate the error on the single 224 224 center crop from
an image whose shorter side is 256.
Our models are realized by the form of Fig. 3(c). We
perform batch normalization (BN) [17] right after the con5The marginally smaller number of parameters and marginally higher
FLOPs are mainly caused by the blocks where the map sizes change.volutions in Fig. 3(c).6ReLU is performed right after each
BN, expect for the output of the block where ReLU is performed after the adding to the shortcut, following [14].
We note that the three forms in Fig. 3 are strictly equivalent, when BN and ReLU are appropriately addressed as
mentioned above. We have trained all three forms and
obtained the same results. We choose to implement by
Fig. 3(c) because it is more succinct and faster than the other
two forms.
5. Experiments
5.1. Experiments on ImageNet-1K
We conduct ablation experiments on the 1000-class ImageNet classiﬁcation task [33]. We follow [14] to construct
50-layer and 101-layer residual networks. We simply replace all blocks in ResNet-50/101 with our blocks.
Notations . Because we adopt the two rules in Sec. 3.1, it is


Notations . Because we adopt the two rules in Sec. 3.1, it is
sufﬁcient for us to refer to an architecture by the template.
For example, Table 1 shows a ResNeXt-50 constructed by a
template with cardinality = 32 and bottleneck width =4d
(Fig. 3). This network is denoted as ResNeXt-50 ( 324d)
for simplicity. We note that the input/output width of the
template is ﬁxed as 256-d (Fig. 3), and all widths are doubled each time when the feature map is subsampled (see
Table 1).
Cardinality vs. Width . We ﬁrst evaluate the trade-off between cardinality Cand bottleneck width, under preserved
complexity as listed in Table 2. Table 3 shows the results
and Fig. 5 shows the curves of error vs. epochs. Comparing with ResNet-50 (Table 3 top and Fig. 5 left), the 32 4d
ResNeXt-50 has a validation error of 22.2%, which is 1.7%
lower than the ResNet baseline’s 23.9%. With cardinality C
increasing from 1 to 32 while keeping complexity, the error
rate keeps reducing. Furthermore, the 32 4d ResNeXt also


increasing from 1 to 32 while keeping complexity, the error
rate keeps reducing. Furthermore, the 32 4d ResNeXt also
has a much lower training error than the ResNet counterpart, suggesting that the gains are notfrom regularization
but from stronger representations .
Similar trends are observed in the case of ResNet-101
(Fig. 5 right, Table 3 bottom), where the 32 4d ResNeXt101 outperforms the ResNet-101 counterpart by 0.8%. Although this improvement of validation error is smaller than
that of the 50-layer case, the improvement of training error is still big (20% for ResNet-101 and 16% for 32 4d
ResNeXt-101, Fig. 5 right). In fact, more training data
will enlarge the gap of validation error, as we show on an
ImageNet-5K set in the next subsection.
Table 3 also suggests that with complexity preserved , increasing cardinality at the price of reducing width starts
to show saturating accuracy when the bottleneck width is
6With BN, for the equivalent form in Fig. 3(a), BN is employed after
aggregating the transformations and before adding to the shortcut.
0 1 02 03 04 05 06 07 08 09 0
epochs1520253035404550top-1 error (%)ResNet-50 (1 x 64d) train


0 1 02 03 04 05 06 07 08 09 0
epochs1520253035404550top-1 error (%)ResNet-50 (1 x 64d) train
ResNet-50 (1 x 64d) val
ResNeXt-50 (32 x 4d) train
ResNeXt-50 (32 x 4d) val
0 1 02 03 04 05 06 07 08 09 0
epochs1520253035404550top-1 error (%)ResNet-101 (1 x 64d) train
ResNet-101 (1 x 64d) val
ResNeXt-101 (32 x 4d) train
ResNeXt-101 (32 x 4d) valFigure 5. Training curves on ImageNet-1K. ( Left): ResNet/ResNeXt-50 with preserved complexity ( 4.1 billion FLOPs, 25 million
parameters); ( Right ): ResNet/ResNeXt-101 with preserved complexity ( 7.8 billion FLOPs,44 million parameters).
setting top-1 error (%)
ResNet-50 164d 23.9
ResNeXt-50 240d 23.0
ResNeXt-50 424d 22.6
ResNeXt-50 814d 22.3


ResNeXt-50 424d 22.6
ResNeXt-50 814d 22.3
ResNeXt-50 324d 22.2
ResNet-101 164d 22.0
ResNeXt-101 240d 21.7
ResNeXt-101 424d 21.4
ResNeXt-101 814d 21.3
ResNeXt-101 324d 21.2
Table 3. Ablation experiments on ImageNet-1K. ( Top): ResNet50 with preserved complexity ( 4.1 billion FLOPs); ( Bottom ):
ResNet-101 with preserved complexity ( 7.8 billion FLOPs). The
error rate is evaluated on the single crop of 224 224 pixels.
small. We argue that it is not worthwhile to keep reducing
width in such a trade-off. So we adopt a bottleneck width
no smaller than 4d in the following.
Increasing Cardinality vs. Deeper/Wider . Next we investigate increasing complexity by increasing cardinality C
or increasing depth or width. The following comparison
can also be viewed as with reference to 2 FLOPs of the
ResNet-101 baseline. We compare the following variants


or increasing depth or width. The following comparison
can also be viewed as with reference to 2 FLOPs of the
ResNet-101 baseline. We compare the following variants
that have15 billion FLOPs. (i) Going deeper to 200 layers. We adopt the ResNet-200 [15] implemented in [11].
(ii)Going wider by increasing the bottleneck width. (iii)
Increasing cardinality by doubling C.
Table 4 shows that increasing complexity by 2 consistently reduces error vs. the ResNet-101 baseline (22.0%).
But the improvement is small when going deeper (ResNet200, by 0.3%) or wider (wider ResNet-101, by 0.7%).
On the contrary, increasing cardinality Cshows muchsetting top-1 err (%) top-5 err (%)
1complexity references:
ResNet-101 164d 22.0 6.0
ResNeXt-101 324d 21.2 5.6
2complexity models follow:
ResNet- 200[15] 164d 21.7 5.8
ResNet-101, wider 1100d 21.3 5.7
ResNeXt-101 264d 20.7 5.5
ResNeXt-101 644d 20.4 5.3


ResNeXt-101 264d 20.7 5.5
ResNeXt-101 644d 20.4 5.3
Table 4. Comparisons on ImageNet-1K when the number of
FLOPs is increased to 2 of ResNet-101’s. The error rate is evaluated on the single crop of 224 224 pixels. The highlighted factors
are the factors that increase complexity.
better results than going deeper or wider . The 264d
ResNeXt-101 ( i.e., doubling Con 164d ResNet-101 baseline and keeping the width) reduces the top-1 error by 1.3%
to 20.7%. The 644d ResNeXt-101 ( i.e., doubling Con
324d ResNeXt-101 and keeping the width) reduces the
top-1 error to 20.4%.
We also note that 32 4d ResNet-101 (21.2%) performs
better than the deeper ResNet-200 and the wider ResNet101, even though it has only 50% complexity. This again
shows that cardinality is a more effective dimension than
the dimensions of depth and width.
Residual connections. The following table shows the effects of the residual (shortcut) connections:
setting w/ residual w/o residual


the dimensions of depth and width.
Residual connections. The following table shows the effects of the residual (shortcut) connections:
setting w/ residual w/o residual
ResNet-50 164d 23.9 31.2
ResNeXt-50 324d 22.2 26.1
Removing shortcuts from the ResNeXt-50 increases the error by 3.9 points to 26.1%. Removing shortcuts from its
ResNet-50 counterpart is much worse (31.2%). These comparisons suggest that the residual connections are helpful
foroptimization , whereas aggregated transformations are
stronger representations , as shown by the fact that they
perform consistently better than their counterparts with or
without residual connections.
Performance. For simplicity we use Torch’s built-in
grouped convolution implementation, without special optimization. We note that this implementation was brute-force
and not parallelization-friendly. On 8 GPUs of NVIDIA
M40, training 324d ResNeXt-101 in Table 3 takes 0.95s
per mini-batch, vs. 0.70s of ResNet-101 baseline that has
similar FLOPs. We argue that this is a reasonable overhead.
We expect carefully engineered lower-level implementation
(e.g., in CUDA) will reduce this overhead. We also expect


We expect carefully engineered lower-level implementation
(e.g., in CUDA) will reduce this overhead. We also expect
that the inference time on CPUs will present less overhead.
Training the 2complexity model (64 4d ResNeXt-101)
takes 1.7s per mini-batch and 10 days total on 8 GPUs.
Comparisons with state-of-the-art results. Table 5 shows
more results of single-crop testing on the ImageNet validation set. In addition to testing a 224 224 crop, we
also evaluate a 320 320 crop following [15]. Our results compare favorably with ResNet, Inception-v3/v4, and
Inception-ResNet-v2, achieving a single-crop top-5 error
rate of 4.4%. In addition, our architecture design is much
simpler than all Inception models, and requires considerably fewer hyper-parameters to be set by hand.
ResNeXt is the foundation of our entries to the ILSVRC
2016 classiﬁcation task, in which we achieved 2ndplace.
We note that many models (including ours) start to get saturated on this dataset after using multi-scale and/or multicrop testing. We had a single-model top-1/top-5 error rates


of 17.7%/3.7% using the multi-scale dense testing in [14],
on par with Inception-ResNet-v2’s single-model results of
17.8%/3.7% that adopts multi-scale, multi-crop testing. We
had an ensemble result of 3.03% top-5 error on the test set,
on par with the winner’s 2.99% and Inception-v4/InceptionResNet-v2’s 3.08% [37].
224224 320320 / 299 299
top-1 err top-5 err top-1 err top-5 err
ResNet-101 [14] 22.0 6.0 - ResNet-200 [15] 21.7 5.8 20.1 4.8
Inception-v3 [39] - - 21.2 5.6
Inception-v4 [37] - - 20.0 5.0
Inception-ResNet-v2 [37] - - 19.9 4.9
ResNeXt-101 ( 644d) 20.4 5.3 19.1 4.4
Table 5. State-of-the-art models on the ImageNet-1K validation


Table 5. State-of-the-art models on the ImageNet-1K validation
set (single-crop testing). The test size of ResNet/ResNeXt is
224224 and 320320 as in [15] and of the Inception models
is 299299.
12345
mini-batches ×1052025303540455055top-1 error (%)ResNet-101 (1 x 64d) val
ResNeXt-101 (32 x 4d) valFigure 6. ImageNet-5K experiments. Models are trained on the
5K set and evaluated on the original 1K validation set, plotted as
a 1K-way classiﬁcation task. ResNeXt and its ResNet counterpart
have similar complexity.
5K-way classiﬁcation 1K-way classiﬁcation
setting top-1 top-5 top-1 top-5
ResNet-50 164d 45.5 19.4 27.1 8.2
ResNeXt-50 324d 42.3 16.8 24.4 6.6
ResNet-101 164d 42.4 16.9 24.2 6.8
ResNeXt-101 324d 40.1 15.1 22.2 5.7


ResNeXt-101 324d 40.1 15.1 22.2 5.7
Table 6. Error (%) on ImageNet-5K . The models are trained on
ImageNet-5K and tested on the ImageNet-1K val set , treated as a
5K-way classiﬁcation task or a 1K-way classiﬁcation task at test
time. ResNeXt and its ResNet counterpart have similar complexity. The error is evaluated on the single crop of 224 224 pixels.
5.2. Experiments on ImageNet-5K
The performance on ImageNet-1K appears to saturate.
But we argue that this is not because of the capability of the
models but because of the complexity of the dataset. Next
we evaluate our models on a larger ImageNet subset that
has 5000 categories.
Our 5K dataset is a subset of the full ImageNet-22K set
[33]. The 5000 categories consist of the original ImageNet1K categories and additional 4000 categories that have the
largest number of images in the full ImageNet set. The 5K
set has 6.8 million images, about 5 of the 1K set. There is
no ofﬁcial train/val split available, so we opt to evaluate on


set has 6.8 million images, about 5 of the 1K set. There is
no ofﬁcial train/val split available, so we opt to evaluate on
the original ImageNet-1K validation set. On this 1K-class
val set, the models can be evaluated as a 5K-way classiﬁcation task (all labels predicted to be the other 4K classes are
automatically erroneous) or as a 1K-way classiﬁcation task
(softmax is applied only on the 1K classes) at test time.
The implementation details are the same as in Sec. 4.
The 5K-training models are all trained from scratch, and
# params CIFAR-10 CIFAR-100
Wide ResNet [43] 36.5M 4.17 20.50
ResNeXt-29, 864d 34.4M 3.65 17.77
ResNeXt-29, 1664d 68.1M 3.58 17.31
Table 7. Test error (%) and model size on CIFAR. Our results are
the average of 10 runs.
are trained for the same number of mini-batches as the 1Ktraining models (so 1/5 epochs). Table 6 and Fig. 6 show


are trained for the same number of mini-batches as the 1Ktraining models (so 1/5 epochs). Table 6 and Fig. 6 show
the comparisons under preserved complexity. ResNeXt-50
reduces the 5K-way top-1 error by 3.2% comparing with
ResNet-50, and ResNetXt-101 reduces the 5K-way top-1
error by 2.3% comparing with ResNet-101. Similar gaps
are observed on the 1K-way error. These demonstrate the
stronger representational power of ResNeXt.
Moreover, we ﬁnd that the models trained on the 5K
set (with 1K-way error 22.2%/5.7% in Table 6) perform
competitively comparing with those trained on the 1K set
(21.2%/5.6% in Table 3), evaluated on the same 1K-way
classiﬁcation task on the validation set. This result is
achieved without increasing the training time (due to the
same number of mini-batches) and without ﬁne-tuning. We
argue that this is a promising result, given that the training
task of classifying 5K categories is a more challenging one.
5.3. Experiments on CIFAR


argue that this is a promising result, given that the training
task of classifying 5K categories is a more challenging one.
5.3. Experiments on CIFAR
We conduct more experiments on CIFAR-10 and 100
datasets [23]. We use the architectures as in [14] and replace the basic residual block by the bottleneck template
of2
6411, 64
33, 64
11, 2563
75. Our networks start with a single 3 3 conv
layer, followed by 3 stages each having 3 residual blocks,
and end with average pooling and a fully-connected classiﬁer (total 29-layer deep), following [14]. We adopt the same
translation and ﬂipping data augmentation as [14]. Implementation details are in the appendix.
We compare two cases of increasing complexity based
on the above baseline: (i) increase cardinality and ﬁx all
widths, or (ii) increase width of the bottleneck and ﬁx cardinality = 1. We train and evaluate a series of networks
under these changes. Fig. 7 shows the comparisons of test
error rates vs. model sizes. We ﬁnd that increasing cardinality is more effective than increasing width, consistent to
what we have observed on ImageNet-1K. Table 7 shows the


what we have observed on ImageNet-1K. Table 7 shows the
results and model sizes, comparing with the Wide ResNet
[43] which is the best published record. Our model with a
similar model size (34.4M) shows results better than Wide
ResNet. Our larger method achieves 3.58% test error (average of 10 runs) on CIFAR-10 and 17.31% on CIFAR-100.
To the best of our knowledge, these are the state-of-the-art
results (with similar data augmentation) in the literature including unpublished technical reports.
4 8 16 32 64 128
# of parameters (M)3.53.63.73.83.94.04.14.24.34.44.5test error (%)ResNet-29 (increase width)
ResNeXt-29 (increase cardinality)1x64d
16x64d8x64d4x64d2x64d
1x384d1x128d
1x256d1x192dFigure 7. Test error vs. model size on CIFAR-10. The results are
computed with 10 runs, shown with standard error bars. The labels
show the settings of the templates.
setting AP@0.5 AP
ResNet-50 164d 47.6 26.5


show the settings of the templates.
setting AP@0.5 AP
ResNet-50 164d 47.6 26.5
ResNeXt-50 324d 49.7 27.5
ResNet-101 164d 51.1 29.8
ResNeXt-101 324d 51.9 30.0
Table 8. Object detection results on the COCO minival set.
ResNeXt and its ResNet counterpart have similar complexity.
5.4. Experiments on COCO object detection
Next we evaluate the generalizability on the COCO object detection set [27]. We train the models on the 80k training set plus a 35k val subset and evaluate on a 5k val subset
(called minival ), following [1]. We evaluate the COCOstyle Average Precision (AP) as well as AP@IoU=0.5 [27].
We adopt the basic Faster R-CNN [32] and follow [14] to
plug ResNet/ResNeXt into it. The models are pre-trained
on ImageNet-1K and ﬁne-tuned on the detection set. Implementation details are in the appendix.


on ImageNet-1K and ﬁne-tuned on the detection set. Implementation details are in the appendix.
Table 8 shows the comparisons. On the 50-layer baseline, ResNeXt improves AP@0.5 by 2.1% and AP by 1.0%,
without increasing complexity. ResNeXt shows smaller improvements on the 101-layer baseline. We conjecture that
more training data will lead to a larger gap, as observed on
the ImageNet-5K set.
It is also worth noting that recently ResNeXt has been
adopted in Mask R-CNN [12] that achieves state-of-the-art
results on COCO instance segmentation and object detection tasks.
Acknowledgment
S.X. and Z.T.’s research was partly supported by NSF
IIS-1618477. The authors would like to thank Tsung-Yi
Lin and Priya Goyal for valuable discussions.
A. Implementation Details: CIFAR
We train the models on the 50k training set and evaluate
on the 10k test set. The input image is 32 32 randomly
cropped from a zero-padded 40 40 image or its ﬂipping,
following [14]. No other data augmentation is used. The


cropped from a zero-padded 40 40 image or its ﬂipping,
following [14]. No other data augmentation is used. The
ﬁrst layer is 33 conv with 64 ﬁlters. There are 3 stages
each having 3 residual blocks, and the output map size is
32, 16, and 8 for each stage [14]. The network ends with a
global average pooling and a fully-connected layer. Width
is increased by 2when the stage changes (downsampling),
as in Sec. 3.1. The models are trained on 8 GPUs with a
mini-batch size of 128, with a weight decay of 0.0005 and
a momentum of 0.9. We start with a learning rate of 0.1
and train the models for 300 epochs, reducing the learning
rate at the 150-th and 225-th epoch. Other implementation
details are as in [11].
B. Implementation Details: Object Detection
We adopt the Faster R-CNN system [32]. For simplicity
we do not share the features between RPN and Fast R-CNN.
In the RPN step, we train on 8 GPUs with each GPU holding
2 images per mini-batch and 256 anchors per image. We
train the RPN step for 120k mini-batches at a learning rate


2 images per mini-batch and 256 anchors per image. We
train the RPN step for 120k mini-batches at a learning rate
of 0.02 and next 60k at 0.002. In the Fast R-CNN step, we
train on 8 GPUs with each GPU holding 1 image and 64
regions per mini-batch. We train the Fast R-CNN step for
120k mini-batches at a learning rate of 0.005 and next 60k at
0.0005, We use a weight decay of 0.0001 and a momentum
of 0.9. Other implementation details are as in https://
github.com/rbgirshick/py-faster-rcnn .
References
[1] S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Insideoutside net: Detecting objects in context with skip
pooling and recurrent neural networks. In CVPR ,
2016.
[2] G. Cantor. ¨Uber unendliche, lineare punktmannichfaltigkeiten, arbeiten zur mengenlehre aus den jahren
1872-1884. 1884.
[3] R. Collobert, S. Bengio, and J. Mari ´ethoz. Torch: a


1872-1884. 1884.
[3] R. Collobert, S. Bengio, and J. Mari ´ethoz. Torch: a
modular machine learning software library. Technical
report, Idiap, 2002.
[4] A. Conneau, H. Schwenk, L. Barrault, and Y . Lecun. Very deep convolutional networks for natural language processing. arXiv:1606.01781 , 2016.
[5] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR , 2005.
[6] E. Denton, W. Zaremba, J. Bruna, Y . LeCun, and
R. Fergus. Exploiting linear structure within convolutional networks for efﬁcient evaluation. In NIPS ,
2014.[7] J. Donahue, Y . Jia, O. Vinyals, J. Hoffman, N. Zhang,
E. Tzeng, and T. Darrell. Decaf: A deep convolutional
activation feature for generic visual recognition. In
ICML , 2014.


E. Tzeng, and T. Darrell. Decaf: A deep convolutional
activation feature for generic visual recognition. In
ICML , 2014.
[8] D. Eigen, J. Rolfe, R. Fergus, and Y . LeCun. Understanding deep architectures using a recursive convolutional network. arXiv:1312.1847 , 2013.
[9] R. Girshick. Fast R-CNN. In ICCV , 2015.
[10] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich
feature hierarchies for accurate object detection and
semantic segmentation. In CVPR , 2014.
[11] S. Gross and M. Wilber. Training and investigating Residual Nets. https://github.com/
facebook/fb.resnet.torch , 2016.
[12] K. He, G. Gkioxari, P. Doll ´ar, and R. Girshick. Mask
R-CNN. arXiv:1703.06870 , 2017.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep
into rectiﬁers: Surpassing human-level performance
on imagenet classiﬁcation. In ICCV , 2015.


into rectiﬁers: Surpassing human-level performance
on imagenet classiﬁcation. In ICCV , 2015.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual
learning for image recognition. In CVPR , 2016.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV , 2016.
[16] Y . Ioannou, D. Robertson, R. Cipolla, and A. Criminisi. Deep roots: Improving cnn efﬁciency with hierarchical ﬁlter groups. arXiv:1605.06489 , 2016.
[17] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal
covariate shift. In ICML , 2015.
[18] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank
expansions. In BMVC , 2014.
[19] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,


[19] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,
R. Girshick, S. Guadarrama, and T. Darrell. Caffe:
Convolutional architecture for fast feature embedding.
arXiv:1408.5093 , 2014.
[20] N. Kalchbrenner, L. Espeholt, K. Simonyan, A. v. d.
Oord, A. Graves, and K. Kavukcuoglu. Neural machine translation in linear time. arXiv:1610.10099 ,
2016.
[21] Y .-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and
D. Shin. Compression of deep convolutional neural
networks for fast and low power mobile applications.
InICLR , 2016.
[22] P. Kontschieder, M. Fiterau, A. Criminisi, and S. R.
Bul`o. Deep convolutional neural decision forests. In
ICCV , 2015.
[23] A. Krizhevsky. Learning multiple layers of features
from tiny images. Tech Report , 2009.


ICCV , 2015.
[23] A. Krizhevsky. Learning multiple layers of features
from tiny images. Tech Report , 2009.
[24] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classiﬁcation with deep convolutional neural
networks. In NIPS , 2012.
[25] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition.
Neural computation , 1989.
[26] M. Lin, Q. Chen, and S. Yan. Network in network. In
ICLR , 2014.
[27] T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona,
D. Ramanan, P. Doll ´ar, and C. L. Zitnick. Microsoft
COCO: Common objects in context. In ECCV . 2014.
[28] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR ,
2015.


[28] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR ,
2015.
[29] D. G. Lowe. Distinctive image features from scaleinvariant keypoints. IJCV , 2004.
[30] A. Oord, S. Dieleman, H. Zen, K. Simonyan,
O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior,
and K. Kavukcuoglu. Wavenet: A generative model
for raw audio. arXiv:1609.03499 , 2016.
[31] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning
to segment object candidates. In NIPS , 2015.
[32] S. Ren, K. He, R. Girshick, and J. Sun. Faster RCNN: Towards real-time object detection with region
proposal networks. In NIPS , 2015.
[33] O. Russakovsky, J. Deng, H. Su, J. Krause,
S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla,
M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet


M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet
Large Scale Visual Recognition Challenge. IJCV ,
2015.
[34] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional
networks. In ICLR , 2014.
[35] L. Sifre and S. Mallat. Rigid-motion scattering for
texture classiﬁcation. arXiv:1403.1687 , 2014.
[36] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In
ICLR , 2015.
[37] C. Szegedy, S. Ioffe, and V . Vanhoucke. Inceptionv4, inception-resnet and the impact of residual connections on learning. In ICLR Workshop , 2016.
[38] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR ,
2015.


D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR ,
2015.
[39] C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and
Z. Wojna. Rethinking the inception architecture for
computer vision. In CVPR , 2016.[40] A. Veit, M. Wilber, and S. Belongie. Residual networks behave like ensembles of relatively shallow network. In NIPS , 2016.
[41] Y . Wu, M. Schuster, Z. Chen, Q. V . Le, M. Norouzi,
W. Macherey, M. Krikun, Y . Cao, Q. Gao,
K. Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and
machine translation. arXiv:1609.08144 , 2016.
[42] W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer,
A. Stolcke, D. Yu, and G. Zweig. The Microsoft
2016 Conversational Speech Recognition System.
arXiv:1609.03528 , 2016.


2016 Conversational Speech Recognition System.
arXiv:1609.03528 , 2016.
[43] S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC , 2016.
[44] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional neural networks. In ECCV ,
2014.


Deep Watershed Transform for Instance Segmentation
Min Bai Raquel Urtasun
Department of Computer Science, University of Toronto
fmbai, urtasun g@cs.toronto.edu
Abstract
Most contemporary approaches to instance segmentation use complex pipelines involving conditional random
ﬁelds, recurrent neural networks, object proposals, or template matching schemes. In this paper, we present a simple yet powerful end-to-end convolutional neural network to
tackle this task. Our approach combines intuitions from the
classical watershed transform and modern deep learning to
produce an energy map of the image where object instances
are unambiguously represented as energy basins. We then
perform a cut at a single energy level to directly yield connected components corresponding to object instances. Our
model achieves more than double the performance over
the state-of-the-art on the challenging Cityscapes Instance
Level Segmentation task.
1. Introduction
Instance segmentation seeks to identify the semantic
class of each pixel as well as associate each pixel with a
physical instance of an object. This is in contrast with
semantic segmentation, which is only concerned with the
ﬁrst task. Instance segmentation is particularly challenging in street scenes, where the scale of the objects can vary
tremendously. Furthermore, the appearance of objects are

Summary:
Most contemporary approaches to instance segmentation use complex pipelines
involving conditional random fields, recurrent neural networks, object
proposals, or template matching schemes. In our paper, we present a simple yet
powerful end-to-end convolutional neural network to tackle this task. Our
approach combines intuitions from the classical watershed transform and modern
deep learning to produce an energy map of the image where object instances are
unambiguously represented as basins in the energy map. We then perform a cut at
a single energy level to directly yield connected components corresponding to
object instances. Our model more than doubles the performance of the
state-of-the-art on the challenging Cityscapes Instance Level Segmentation
task.


ﬁrst task. Instance segmentation is particularly challenging in street scenes, where the scale of the objects can vary
tremendously. Furthermore, the appearance of objects are
affected by partial occlusions, specularities, intensity saturation, and motion blur. Solving this task will, however,
tremendously beneﬁt applications such as object manipulation in robotics, or scene understanding and tracking for
self-driving cars.
Current approaches generally use complex pipelines to
handle instance extraction involving object proposals [20,
22, 7], conditional random ﬁelds (CRF) [32, 33], large recurrent neural networks (RNN) [24, 23, 2], or template
matching [28]. In contrast, we present an exceptionally
simple and intuitive method that signiﬁcantly outperforms
the state-of-the-art. In particular, we derive a novel approach which brings together classical grouping techniques
and modern deep neural networks.
The watershed transform is a well studied method inmathematical morphology. Its application to image segmentation can be traced back to the 70’s [4, 3]. The idea
behind this transform is fairly intuitive. Any greyscale image can be considered as a topographic surface. If we ﬂood
this surface from its minima and prevent the merging of the


behind this transform is fairly intuitive. Any greyscale image can be considered as a topographic surface. If we ﬂood
this surface from its minima and prevent the merging of the
waters coming from different sources, we effectively partition the image into different components (i.e., regions).
This transformation is typically applied to the image gradient, thus the basins correspond to homogeneous regions in
the image. A signiﬁcant limitation of the watershed transform is its propensity to over-segment the image. One of
the possible solutions is to estimate the locations of object
instance markers, which guide the selection of a subset of
these basins [11, 21]. Heuristics on the relative depth of the
basins can be exploited in order to merge basins. However,
extracting appropriate markers and creating good heuristics
is difﬁcult in practice. As a consequence, modern techniques for instance segmentation do not exploit the watershed transform.
In this paper, we propose a novel approach which combines the strengths of modern deep neural networks with the
power of this classical bottom-up grouping technique. We
propose to directly learn the energy of the watershed transform such that each basin corresponds to a single instance,
while all dividing ridges are at the same height in the energy domain. As a consequence, the components can be


while all dividing ridges are at the same height in the energy domain. As a consequence, the components can be
extracted by a cut at a single energy level without leading
to over-segmentation. Our approach has several key advantages: it can be easily trained end-to-end, and produces very
fast and accurate estimates. Our method does not rely on iterative strategies such as RNNs, thus has a constant runtime
regardless of the number of object instances.
We demonstrate the effectiveness of our approach in the
challenging Cityscapes Instance Segmentation benchmark
[6], and show that we more than double the performance
of the current state-of-the-art. In the following sections, we
ﬁrst review related work. We then present the details behind
our intuition and model design, followed by an analysis of
our model’s performance. Finally, we explore the impact of
various parts of our model in ablation studies.
1arXiv:1611.08303v2  [cs.CV]  4 May 2017
(a) Input Image
 (b) Semantic Segmentation [34]
 (c) Our Instance Segmentation
 (d) GT Instance Segmentation
Figure 1: Sample prediction: the input image is gated by sem. segmentation from [34] and passed through our model.


(d) GT Instance Segmentation
Figure 1: Sample prediction: the input image is gated by sem. segmentation from [34] and passed through our model.
2. Related
Several instance level segmentation approaches have
been proposed in recent years. We now brieﬂy review them.
Proposal based: Many approaches are based on the reﬁnement of object proposals. For example, [1] generates
object segments proposals, and reasons about combining
them into object instances. In a similar spirit, [12] selects
proposals using CNN features and non-maximum suppression. Based on this, [5] further reasons about multiple object proposals to handle occlusion scenarios where single
objects are split into multiple disconnected patches. [7] uses
a deep cascaded neural network to propose object instance
bounding boxes, reﬁne instance masks, and semantically
label the masks in sequence. [20, 22] generate segmentation proposals using deep CNNs, which are then further reﬁned to achieve better segmentation boundaries. Additionally, [31] uses a modiﬁed R-CNN model to propose instance
bounding boxes, which are then further reﬁned to obtain instance level segmentation.


bounding boxes, which are then further reﬁned to obtain instance level segmentation.
Deep structured models: [32, 33] ﬁrst use CNNs to perform local instance disambiguation and labelling, followed
by a global conditional random ﬁeld (CRF) to achieve instance label consistency. Recent work by [2] uses object
detection proposals in conjunction with a deep high order
CRF to reason about pixel assignment in overlapping object
proposal boxes.
Template matching: [28] extracts image features using
CNNs to assign a sector label to each pixel in an object instance, which corresponds to one of eight discretized radial
bins around the object’s visible center. A template matching
scheme is then used to associate instance center proposals
and pixels with an object instance.
Recurrent Networks: [24] uses CNNs for feature extraction, followed by a recurrent neural network (RNN) that
generates instance labels for one object at a time. The recurrent structures (based on ConvLSTM [25]) keep track of
instances that have already been generated, and inhibit these
regions from further instance generation. Additionally, [23]
extracts image features similar to [28] and employs a fairly
complex pipeline including a ConvLSTM structure to direct


regions from further instance generation. Additionally, [23]
extracts image features similar to [28] and employs a fairly
complex pipeline including a ConvLSTM structure to direct
a bounding box generation network followed by a segmentation network that extracts individual instances.CNN: [15] leverages only a CNN trained to provide multiple outputs to simultaneously predict instance numbers,
bounding box coordinates, and category conﬁdence scores
for each pixel. This is followed by generic clustering algorithms to group the resulting output into instance-wise labels. Additionally, [13] proposed deep convolutional neural
network that learns the underlying shapes of objects, and
performs multiple unconstrained inference steps to reﬁne
regions corresponding to an object instance while ignoring
neighboring pixels that do not belong to the primary object.
Proposal + recursion: [14] proposed a novel method
that recursively reﬁnes proposals.
In contrast, in this paper we propose a novel approach
which combines the strengths of modern deep neural networks with the power of the watershed transform. Our
model is simple, fast, accurate, and inherently handles an
arbitrary number of instances per image with ease.
3. A Review on the Watershed Transform
We start our discussion with a review of the watershed


arbitrary number of instances per image with ease.
3. A Review on the Watershed Transform
We start our discussion with a review of the watershed
transform, a well studied method in mathematical morphology. This technique is built on the fact that any greyscale
image can be considered as a topographic surface. If we
ﬂood this surface from its minima while building barriers
to prevent the merging of the waters coming from different sources, we effectively partition the image into different
components or regions. These components are called catchment basins. The barriers or watershed lines then represent
the boundaries between the different basins (i.e., boundaries
between regions).
This process is illustrated in the ﬁrst row of Fig. 2 for
a one dimensional energy function. In this case, the watershed transform results in seven components, which are
illustrated in different colors. Note that the traditional watershed transform tends to produce an over-segmentation of
the image due to spurious small ridges which produce separate components. In this example, although there are 3
main components, the watershed transform over-segments
the image because of small perturbations in the energy.
Several algorithms have been developed to estimate the
components. [3] proposed an algorithm that iteratively ﬁlls
the watershed landscape from each local minimum, adding
dams wherever two neighbouring bodies of water meet.


components. [3] proposed an algorithm that iteratively ﬁlls
the watershed landscape from each local minimum, adding
dams wherever two neighbouring bodies of water meet.
(a) Traditional Watershed Energy
(b) Our learned energy
Figure 2: Conceptual 1-D comparison between the traditional watershed transform and our deep watershed transform. The resulting instances are represented as colors
across the top.
These dams deﬁne the segmentation boundaries. Additionally, [18] details a number of alternative watershed transform algorithms, including topological distance, shortest
path algorithms, spanning trees, and marker based methods.
The watershed transform is typically applied to the image gradient, while the catchment basins correspond to homogeneous grey level regions in the image. However, estimating sharp gradients that represent the boundaries between different instances is a very challenging process.
In the next section, we will show an alternative approach
which directly learns to predict the energy landscape.
4. Deep Watershed Tranform
In this section, we present our novel approach to instance
level segmentation. In particular, we learn the energy of the
watershed transform with a feed-forward neural network.
The idea behind our approach is very intuitive. It consists
of learning to predict an energy landscape such that each
basin corresponds to a single instance, while all ridges are
at the same height in the energy domain. As a consequence,


of learning to predict an energy landscape such that each
basin corresponds to a single instance, while all ridges are
at the same height in the energy domain. As a consequence,
the watershed cut corresponds to a single threshold in the
energy, which does not lead to over segmentation. We refer
the reader to the lower half of Fig. 2 for an illustration of
the desired energy.
Unfortunately, learning the energy landscape from
scratch is a complex task. Therefore, we aid the network
by deﬁning an intermediate task, where we learn the direction of descent of the watershed energy. This is then passedthrough another set of network layers to learn the ﬁnal energy. In principle, one can interpret this network as learning
to perform the distance transform of each point within an
object instance to the instance’s boundary. Fig. 4 shows an
example of the input, intermediate results, and ﬁnal output
of our method. We refer the reader to Fig. 3 for an illustration of our network architecture.
4.1. Direction Network (DN)
Our approach leverages semantic segmentation as input
to focus only on relevant areas of the image. Note that our
network is agnostic to the choice of segmentation algorithm.
In our paper, we use the semantic segmentation results from
PSPNet [34].


network is agnostic to the choice of segmentation algorithm.
In our paper, we use the semantic segmentation results from
PSPNet [34].
The network takes as input the original RGB image gated
by a binarized semantic segmentation, where all pixels that
are not part of one of the semantic classes of interest are set
to zero. The input image is augmented by adding the semantic segmentation as a fourth channel. Because the RGB
values range from 0 to 255 for each channel before mean
subtraction, we likewise scale the encoding of the semantic
segmentation image such that the class labels are equally
spaced numerically, and the variance is approximately equal
to that of values in the RGB channels.
To aid the model in producing an accurate energy landscape, we pre-train the overall network’s ﬁrst part (referenced here as the Direction Network, DN) to estimate the
direction of descent of the energy at each pixel. We parameterize it with a unit vector pointing away from the nearest
point on the object’s instance boundary. This supervision
gives a very strong training signal: the direction of the nearest boundary is directly encoded in the unit vector. Furthermore, a pair of pixels straddling an occlusion boundary
between two objects will have target unit vectors pointing
in opposite directions. Here, associating a pixel with the


between two objects will have target unit vectors pointing
in opposite directions. Here, associating a pixel with the
wrong object instance incurs the maximum possible angular
error. This is true regardless of the shape of the objects, even
when they are highly concave or elongated. This forces the
network to learn very accurate boundary localization at the
pixel level.
LetDgt(p)be the ground truth distance transform from
a point to the boundary of the instance it belongs to. We
deﬁne our ground truth targets as the normalized gradient
~ upof this distance transform. More formally,
~ up;gt=rDgt(p)
jrDgt(p)j
Thus, the DN produces a two channel output at the input
image’s resolution representing the directional unit vector.
It is important to note that the normalization layer at the output of the DN restricts the sum of each channel’s squared
output to be 1. This greatly reduces the difﬁculty of using the output of non-linear activation functions to model
Figure 3: Complete network architecture. The network takes the original RGB image gated by semantic segmentation and
concatenated with the semantic segmentation as input, and produces the deep watershed transform energy map as the output.
(a) Input Image
 (b) GT angle of ~ up
 (c) GT Watershed Energy


(a) Input Image
 (b) GT angle of ~ up
 (c) GT Watershed Energy
 (d) GT Instances
(e) Sem. Segmentation of [34]
 (f) Pred. angle of ~ up
 (g) Pred. Watershed Transform
 (h) Pred. Instances
Figure 4: Our network takes the RGB image (a) and the semantic segmentation (e) as input, and predicts a unit vector at each
foreground pixel pointing directly away from the nearest boundary (f). Based on this, we then predict a modiﬁed watershed
transform energy (g), upon which we perform cut at a ﬁxed threshold to yield the ﬁnal predictions (h).
a scalar target. Additionally, phrasing the training target as
a unit vector instead of an angle bypasses the problem of
the equivalent angles 0and 360having a large numerical
difference.
The feature extraction portion of the DN’s architecture
is inspired by VGG16 [27]. However, there are several important modiﬁcations. Direction prediction is a very precise task, as the output can vary greatly between neighboring pixels. Thus, it is critical to avoid losing spatial
resolution. We utilize a modiﬁed version of the ﬁrst 13


resolution. We utilize a modiﬁed version of the ﬁrst 13
layers of VGG, where the third and fourth pooling layers are changed to average pooling while the ﬁfth pooling
layer is removed. To preserve spatial resolution, we exploit a high-capacity, multi-scale information aggregation
scheme inspired by popular methods in semantic segmenta-tion [9, 30, 16]. In particular, the outputs of conv3 ,conv4 ,
andconv5 individually undergo a 55convolution followed
by two 11convolutions. After this, the outputs of the
second and third paths are upsampled to the resolution of
the ﬁrst path. The resulting feature volumes from the three
paths are concatenated. This undergoes an additional set
of three 11convolutions, before being upsampled to the
input resolution.
4.2. Watershed Transform Network (WTN)
The second half of the overall network takes as input the
2-channel unit vector map, and produces a discretized modiﬁed watershed transform map with K= 16 possible energy
values. In this case, bin 0 corresponds to background or regions within 2 pixels of an instance boundary and is referred
to as having an energy level of 0. Meanwhile, higher numbered bins with higher energy levels correspond to regions


to as having an energy level of 0. Meanwhile, higher numbered bins with higher energy levels correspond to regions
in the interior of object instances. The bins are chosen to
maximize the binning resolution of energy levels near zero
(to facilitate accurate cutting), while achieving an approximate balance between the total numbers of pixels within
each class.
The WTN is a fairly generic CNN, with an emphasis on
reasoning with high spatial accuracy and resolution. The architecture is shown in the right half of Fig. 3. In particular,
the network consists of two 55convolutional ﬁlter blocks
each followed by a 22average pooling, before undergoing a set of 11convolutions and upsampling to the input
image resolution.
4.3. Network Training
We ﬁrst pre-train our DN and WTN networks. This process is followed by end-to-end training of the whole network.
Direction Network pre-training: We pre-train the network using the mean squared error in the angular domain as
loss:
ldirection =X
p2P objwpkcos1< ~ up;GT; ~ up;pred>k2


loss:
ldirection =X
p2P objwpkcos1< ~ up;GT; ~ up;pred>k2
wherePobjis the set of all pixels belonging to a semantic class with object instances, and wpis a weighting factor proportional to the inverse of the square root of the object instance’s area. We use the mean squared angular error as opposed to the commonly used cosine distance, as
large angular errors (occuring when the network associates
a pixel with the wrong object instance) should incur significant larger penalties than small angular errors (which have
little impact).
This network is pre-trained using the original input images gated by the ground truth semantic segmentation and
the ground truth unit vectors ~ uGT. The lower layers (conv1
to conv5) are initialized with VGG16 [27]. The weights of
the upper layers are initialized randomly according to the
Xavier initialization scheme [10]. However, the intialization variance of the weights of the ﬁnal layer of each prediction branch is set manually such that the variance of each
branch’s output is of the same order of magnitude before
concatenation. This encourages the network to consider the
output of each branch. We use the ADAM optimizer to train


concatenation. This encourages the network to consider the
output of each branch. We use the ADAM optimizer to train
the network for 20 epochs with a batch size of 4, with a constant learning rate of 1e-5 and L2 weight penalty of 1e-5.
Watershed Network pre-training: The network is
trained using a modiﬁed cross-entropy loss
lwatershed =X
p2P objKX
k=1wpck(tp;klog yp;k+tp;klogyp;k)where tp;kis the k-th element of pixel p’s one-hot target
vector, yp;kis the k-th channel of the network output at p,
wpis a coefﬁcient to adjust the importance of smaller objects (as deﬁned in the DN pre-training section), and ckis
a scaling constant speciﬁc to each discretization class. Because the single energy level cut occurs at just above zero
energy (i.e., near object instance boundaries), an accurate
estimation of pixels at low energy levels is of crucial importance. Thus, the set fckgis selected to be in increasing order. In this case, any errors of low energy levels are assigned
a greater level of penalty, which encourages the network to


a greater level of penalty, which encourages the network to
focus on boundary regions. We pre-train the network using
the ground truth semantic segmentation and ground truth
direction predictions as input, and the discretized ground
truth modiﬁed watershed transform as target. All weights
are initialized using Xavier initialization [10]. We train the
network for 25 epochs using the ADAM optimizer. A batch
size of 6, constant learning rate of 5e-4, and a L2 weight
penalty of 1e-6 are used.
End-to-end ﬁne-tuning: We cascaded the pre-trained
models for the DN and WTN and ﬁne-tuned the complete
model for 20 epochs using the RGB image and semantic
segmentation output of PSPNet as input, and the ground
truth distance transforms as the training target. We use a
batch size of 3, constant learning rate of 5e-6, and a L2
weight penalty of 1e-6.
4.4. Energy Cut and Instance Extraction
We cut our watershed transform output at energy level 1
(0 to 2 pixels from boundary) for classes with many smaller
objects ( person ,rider ,bicycle ,motorcycle ), and level 2 (3 to
4 pixels from boundary) for classes with larger objects ( car,


objects ( person ,rider ,bicycle ,motorcycle ), and level 2 (3 to
4 pixels from boundary) for classes with larger objects ( car,
bus,truck ,train ). Following this, the instances are dilated
using a circular structuring element whose radius equals the
maximum erosion at the energy level of the cut. This offsets the boundary erosion from the non-zero threshold. The
connected components are identiﬁed in the resulting image,
directly yielding the proposed instances. The proposals are
further reﬁned by basic hole-ﬁlling. Finally, we remove
small, spurious instances.
5. Experimental Evaluation
In this section, we evaluate our approach on the challenging Cityscapes Instance-Level Semantic Labelling Task [6].
The ofﬁcial benchmark test and validation set results are
found in Tables 1 and 2. We then perform ablation studies with the validation set to examine the performance of
various aspects of our model.
Dataset: The Cityscapes Instance Labelling Task contains 5000 ﬁnely annotated street scene images taken by a
vehicle-mounted camera. Each image has a resolution of
MethodCityscapes Test Set Cityscapes Val Set
AP AP 50% AP 100m AP 50m AP muCov


vehicle-mounted camera. Each image has a resolution of
MethodCityscapes Test Set Cityscapes Val Set
AP AP 50% AP 100m AP 50m AP muCov
van den Brand et al. [29] 2.3% 3.7% 3.9% 4.9% - Cordts et al. [6] 4.6% 12.9% 7.7% 10.3% - Uhrig et al. [28] 8.9% 21.1% 15.3% 16.7% 9.9% Ours 19.4% 35.3% 31.4% 36.8% 21.2% 68.0%
Table 1: Cityscapes instance segmentation results using metrics deﬁned in [6] for AP and [26] for muCov.
Method Person Rider Car Truck Bus Train Motorcycle Bicycle
van den Brand et al. [29] - - 18.2% - - - - Cordts et al. [6] 1.3% 0.6% 10.5% 6.1% 9.7% 5.9% 1.7% 0.5%
Uhrig et al. [28] 12.5% 11.7% 22.5% 3.3% 5.9% 3.2% 6.9% 5.1%


Uhrig et al. [28] 12.5% 11.7% 22.5% 3.3% 5.9% 3.2% 6.9% 5.1%
Ours 15.5% 14.1% 31.5% 22.5% 27.0% 22.9% 13.9% 8.0%
Table 2: Cityscapes instance segmentation class speciﬁc test set AP scores using metrics deﬁned in [6].
2048x1024 pixels. Unlike other commonly used datasets
for instance segmentation (e.g., Pascal VOC [8], BSDS500
[17], and CVPPP [19]) Cityscapes has a large number of
scenes involving dozens of instances with large degrees of
occlusions at vastly different scales. Eight of the semantically labelled categories have instance-level labelling. We
refer the reader to Table 3 for a summary of the statistics of
the object instances in this dataset. Note that while the car
andpeople classes have signiﬁcant numbers of instances,
the other six classes are rather uncommon. As a result, the
rare classes have far less training data, and are thus much
more challenging. We use the ofﬁcial training, validation,
and testing set splits, with 2975, 500, and 1525 images, respectively.


more challenging. We use the ofﬁcial training, validation,
and testing set splits, with 2975, 500, and 1525 images, respectively.
Metric: We report the metrics employed by the
Cityscapes leaderboard. Several variants of the instancelevel average precision score are used. This is calculated
by ﬁnding the precision at various levels of intersectionover-union (IoU) between predicted instances and ground
truth instances, ranging from 50% to 95%. The main score
is the Average Precision (AP). Additionally, there are three
minor scores: AP at 50% overlap, AP of objects closer
than 50m, and AP of objects closer than 100m. Methods
are ranked by the AP score over all classes (mean AP).
Note that AP is a detection score, which does not penalize
overlapping instances or a large number of predictions,
as long as they are ranked in the proper order. Thus,
it places approaches (like ours) which predict a single
instance label (or background) per pixel at a disadvantage,
while favoring detection-based methods. Despite this, we
use these metrics to be consistent with the the evaluation
of the state-of-the-art. For the fbus,truck ,traingclasses,
we order the instances by simply averaging the semantic


of the state-of-the-art. For the fbus,truck ,traingclasses,
we order the instances by simply averaging the semantic
segmentation’s output conﬁdence within each instance to
somewhat counteract the errors by semantic segmentation.Category Total
PixelsTotal
InstancesAverage
SizeInstances
/ Image
Person 6.6e+7 17900 3691 6.0
Rider 7.4e+6 1754 4244 0.6
Car 3.8e+8 26929 14233 9.1
Truck 1.5e+7 482 30648 0.2
Bus 1.3e+7 379 34275 0.1
Train 3.2e+6 168 19143 0.1
Motorcycle 5.4e+6 735 7298 0.2
Bicycle 1.6e+7 3655 4332 1.2
Table 3: Statistics of the training set of the cityscapes instance level segmentation. Average size is in pixels.
Sem. Seg. Src. Sem. Seg. IoU Inst. Seg. AP
LRR [9] 71.8% 20.2%
PSPNet [34] 80.2% 21.2%
Table 4: Comparison of instance segmentation performance
on the validation set with various semantic segmentation


PSPNet [34] 80.2% 21.2%
Table 4: Comparison of instance segmentation performance
on the validation set with various semantic segmentation
sources. Semantic segmentation IoU Class scores [6] are
also provided.
For all other classes, we use a random ordering. Note that
sophisticated ranking techniques can be used to further
improve the score. However, the segmentation quality
remains this same. This again highlights a shortcoming of
using AP as the metric.
In addition to AP, we report the mean weighted coverage
score introduced in [26] on the validation set. This metric
enforces a single instance label per pixel, and is therefore
more suitable for evaluating our approach. We hope that
future work will likewise report this score when applicable.
Method Average Person Rider Car Truck Bus Train Motorcycle Bicycle
Ours 20.50% 15.50% 13.80% 33.10% 23.60% 39.90% 17.40% 11.90% 8.80%
Ours + Ordering 21.24% 15.50% 13.80% 33.10% 27.10% 45.20% 14.50% 11.90% 8.80%


Ours + Oracle Ordering 27.58% 20.60% 18.70% 40.10% 31.50% 50.60% 28.30% 17.40% 13.40%
Table 5: Comparison of AP scores with various instance ordering techniques using the validation set.
Comparison to the state-of-the-art: We show the instance segmentation test set scores in Table 1. Additionally, we show the class-speciﬁc AP scores in Table 2. It is
evident that we achieve a large improvement over the stateof-the-art in all semantic classes. Moreover, we do not use
depth information to train our model, unlike [28].
Analysis of the intermediate training target: Our ﬁnal
network is the result of the end-to-end ﬁne-tuning of two
pre-trained sub-networks (DN and WTN). Fig. 4 (f) shows
the output of the DN after ﬁnetuning. It is evident that the
ﬁne-tuned model retained the direction prediction as an intermediate task. This suggests that the intermediate training
target is effective.
Inﬂuence of semantic segmentation: While we elected
to use PSPNet [34] as our semantic segmentation source, we
additionally demonstrate that our method is able to use other


to use PSPNet [34] as our semantic segmentation source, we
additionally demonstrate that our method is able to use other
sources. Table 4 shows the use of LRR [9] for semantic segmentation. Using the same pre-trained DN and DTN models, we perform end-to-end ﬁne-tuning using LRR as semantic segmentation. We note that the performance of our
model improves with better semantic segmentation. Thus,
future advances in segmentic segmentation methods can
further improve the performance of our approach.
Conﬁdence score estimate: As mentioned, the AP score
calculated by the Cityscapes benchmark requires a conﬁdence score for each instance. For the fbus,truck ,traing
set, we produce a weak ranking based on semantic segmentation softmax conﬁdence. Instances of all other classes are
randomly ranked. Table 5 explores the impact of various ordering schemes. We compare our ordering with random for
all semantic classes, as well as optimal ordering using oracle IoU. We see that ordering using oracle IoU can increase
our model’s performance by 6.34%. Note, however, that
this has no impact on the actual quality of proposed object
instances, which remain the same. This shows the necessity


this has no impact on the actual quality of proposed object
instances, which remain the same. This shows the necessity
of a different metric such as muCov [26] that can evaluate
segmentation-based approaches fairly.
Qualitative Results: Fig. 5 depicts visualizations of sample results on the validation set, which is not used as part of
training. It is evident that our model produces very high
quality instance segmentation results. In these results, predicted and ground truth object instances only share the same
color if they have greater than 50% IoU.Failure Modes: Our model has several weaknesses.
Some images in Fig. 5 demonstrate these cases. The ﬁrst
issue is that the current formulation of our method does not
handle objects that have been separated into multiple pieces
by occlusion. This is most obvious in the 3rd image from
the bottom in Fig. 5 as the far-right vehicle is bisected by the
street sign pole, and the bicycle in the right part of the image just above. The resulting pieces are not merged into one
component. This is a drawback of most bottom-up grouping approaches. The second issue are cases where two objects sharing an occlusion boundary are mostly but not fully
separated by a line of low energy. This is seen in the rightmost vehicle in the 11th image. We anticipate that a combination of our method with top-down reasoning approaches


separated by a line of low energy. This is seen in the rightmost vehicle in the 11th image. We anticipate that a combination of our method with top-down reasoning approaches
will greatly alleviate these two issues.
Because we rely upon correct semantic segmentation, errors from this (such as identifying a train as a bus) cannot be
ﬁxed by our method. This is clearly shown by the truck in
the last example. A possible solution could be to use semantic segmentation as soft gating, or to reason about semantic
and instance segmentation jointly. Finally, some very complex scenes such as some subgroups of people on the left in
the second to fourth example are incorrectly separated by
our model, and are fused together.
6. Conclusion
In this paper, we have proposed a simple instance segmentation technique inspired by the intuitive and classical watershed transform. Using a novel deep convolutional neural network and innovative loss functions for pretraining and ﬁne-tuning, we proposed a model that generates a modiﬁed watershed energy landscape. From this energy landscape, we directly extract high quality object instances. Our experiments show that we can more than double the performance of the state-of-the-art in the challenging Cityscapes Instance Segmentation task. We will release
the network weights and code for training and testing our


the network weights and code for training and testing our
model. In the future, we plan to augment the method to handle object instances bisected by occlusions. Additionally,
we wish to explore the possibility of extending our approach
to perform joint semantic and instance level segmentation,
in hopes of further reﬁning both outputs simultaneously.
Acknowledgements: This work was partially supported
by ONR-N00014-14-1-0232, Samsung, NVIDIA, Google
and NSERC.
(a) Input Image
 (b) Sem. Segmentation [34]
 (c) Our Instance Segmentation
 (d) GT Instance Segmentation
Figure 5: Sample output of our model on the validation set. Note that predicted object instances and ground truth object
instances are only given the same color if they have over 50% IoU.
References
[1] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik. Multiscale Combinatorial Grouping. In CVPR , 2014.
2
[2] A. Arnab and P. Torr. Bottom-up instance segmentation using deep higher-order crfs. In BMVC , 2016. 1, 2
[3] S. Beucher. The watershed transformation applied to image segmentation. Scanning Microscopy International ,


[3] S. Beucher. The watershed transformation applied to image segmentation. Scanning Microscopy International ,
Suppl:6(1):299–314, 1991. 1, 2
[4] S. Beucher and C. Lantuejoul. Use of watersheds in contour
detection. Proc. Int. Workshop Image Processing, Real-Time
Edge and Motion Detection/Estimation , 1976. 1
[5] Y . Chen, X. Liu, and M. Yang. Bottom-up instance segmentation using deep higher-order crfs. In CVPR , 2015. 2
[6] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
Cityscapes Dataset for Semantic Urban Scene Understanding. In CVPR , 2016. 1, 5, 6
[7] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR , 2016. 1,
2
[8] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. The PASCAL Visual Object Classes


and A. Zisserman. The PASCAL Visual Object Classes
Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.
6
[9] C. F. G. Ghiasi. Laplacian Pyramid Reconstruction and Reﬁnement for Semantic Segmentation. In ECCV , 2016. 4, 6,
7
[10] X. Glorot and Y . Bengio. Understanding the difﬁculty of
training deep feedforward neural networks. In AISTATS ,
2010. 5
[11] V . Grau, A. Mewes, M. Alcaiz, R. Kikinis, and S. Warﬁeld.
Improved watershed transform for medical image segmentation using prior information. IEEE Transactions on Medical
Imaging , 23(4):447–458, 2004. 1
[12] B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV , 2014. 2
[13] K. Li, B. Hariharan, and J. Malik. Iterative instance segmentation. In CVPR , 2016. 2


[13] K. Li, B. Hariharan, and J. Malik. Iterative instance segmentation. In CVPR , 2016. 2
[14] X. Liang, Y . Wei, X. Shen, Z. Jie, J. Feng, L. Lin, and S. Yan.
Reversible recursive instance-level object segmentation. In
CVPR , 2016. 2
[15] X. Liang, Y . Wei, X. Shen, J. Yang, L. Lin, and S. Yan. Endto-end instance segmentation and counting with recurrent attention. arXiv preprint arXiv:1509.02636 , 2015. 2
[16] J. Long, E. Shelhamer, and T. Darrell. Fully Convolutional
Networks for Semantic Segmentation. In CVPR , 2015. 4
[17] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database
of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecological statistics. In ICCV , 2001. 6
[18] F. Meyer. The watershed concept and its use in segmentation:
a brief history. arXiv preprint arXiv:1202.0216 , 2012. 3


[18] F. Meyer. The watershed concept and its use in segmentation:
a brief history. arXiv preprint arXiv:1202.0216 , 2012. 3
[19] M. Minervini, A. Fischbach, H.Scharr, , and S. Tsaftaris.
Finely-grained annotated datasets for image-based plant phenotyping. Pattern Recognition Letters , pages 1–10, 2015. 6[20] P. O. Pinheiro, R. Collobert, and P. Dollr. Learning to segment object candidates. In NIPS , 2015. 1, 2
[21] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Dollr. Segmentation of 3d head mr images using morphological reconstruction under constraints and automatic selection of markers. In
ICIP , pages 1075–1078, 2001. 1
[22] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Dollr. Learning
to reﬁne object segments. In ECCV , 2016. 1, 2
[23] M. Ren and R. Zemel. End-to-end instance segmentation and counting with recurrent attention. arXiv preprint


[23] M. Ren and R. Zemel. End-to-end instance segmentation and counting with recurrent attention. arXiv preprint
arXiv:1605.09410 , 2016. 1, 2
[24] B. Romera-Paredes and P. H. S. Torr. Recurrent Instance
Segmentation. In ECCV , 2016. 1, 2
[25] X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo.
Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In NIPS , 2015. 2
[26] N. Silberman, D. Sontag, and R. Fergus. Instance segmentation of indoor scenes using a coverage loss. In ECCV , 2014.
6, 7
[27] K. Simonyan and A. Zisserman. Very Deep Convolutional
Networks for Large-Scale Image Recognition. In ICLR ,
2015. 4, 5
[28] J. Uhrig, M. Cordts, U. Franke, and T. Brox. Pixel-level
Encoding and Depth Layering for Instance-level Semantic
Labeling. In GCPR , 2016. 1, 2, 6, 7


Encoding and Depth Layering for Instance-level Semantic
Labeling. In GCPR , 2016. 1, 2, 6, 7
[29] J. van den Brand, M. Ochs, and R. Mester. Instance-level
Segmentation of Vehicles by Deep Contours. In ACCV Workshop , 2016. 6
[30] F. Yu and V . Koltun. Multi-Scale Context Aggregation by
Dilated Convolutions. In ICLR , 2016. 4
[31] S. Zagoruyko, A. Lerer, T.-Y . Lin, P. O. Pinheiro, S. Gross,
S. Chintala, and P. Doll ´ar. A multipath network for object
detection. In BMVC , 2016. 2
[32] Z. Zhang, S. Fidler, and R. Urtasun. Instance-Level Segmentation with Deep Densely Connected MRFs. In CVPR , 2016.
1, 2
[33] Z. Zhang, A. Schwing, S. Fidler, and R. Urtasun. Monocular
object instance segmentation and depth ordering with cnns.
InICCV , 2015. 1, 2


object instance segmentation and depth ordering with cnns.
InICCV , 2015. 1, 2
[34] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid Scene
Parsing Network. arXiv preprint arXiv:1612.01105 , 2016. 2,
3, 4, 6, 7, 8


Interaction Networks for Learning about Objects,
Relations and Physics
Peter W. Battaglia
Google DeepMind
London, UK N1C 4AG
peterbattaglia@google.comRazvan Pascanu
Google DeepMind
London, UK N1C 4AG
razp@google.comMatthew Lai
Google DeepMind
London, UK N1C 4AG
matthewlai@google.com
Danilo Rezende
Google DeepMind
London, UK N1C 4AG
danilor@google.comKoray Kavukcuoglu
Google DeepMind
London, UK N1C 4AG
korayk@google.com
Abstract
Reasoning about objects, relations, and physics is central to human intelligence, and
a key goal of artiﬁcial intelligence. Here we introduce the interaction network , a
model which can reason about how objects in complex systems interact, supporting
dynamical predictions, as well as inferences about the abstract properties of the
system. Our model takes graphs as input, performs object- and relation-centric
reasoning in a way that is analogous to a simulation, and is implemented using
deep neural networks. We evaluate its ability to reason about several challenging
physical domains: n-body problems, rigid-body collision, and non-rigid dynamics.

Summary:
Reasoning about objects, relations, and physics is central to human
intelligence, and a key goal of artificial intelligence. Here we introduce the
interaction network, a model which can reason about how objects in complex
systems interact, supporting dynamical predictions, as well as inferences about
the abstract properties of the system. Our model takes graphs as input,
performs object- and relation-centric reasoning in a way that is analogous to a
simulation, and is implemented using deep neural networks. We evaluate its
ability to reason about several challenging physical domains: n-body problems,
rigid-body collision, and non-rigid dynamics. Our results show it can be
trained to accurately simulate the physical trajectories of dozens of objects
over thousands of time steps, estimate abstract quantities such as energy, and
generalize automatically to systems with different numbers and configurations
of objects and relations. Our interaction network implementation is the first
general-purpose, learnable physics engine, and a powerful general framework for
reasoning about object and relations in a wide variety of complex real-world
domains.

Peter W. Battaglia
Razvan Pascanu
Matthew Lai
Danilo Rezende
Koray Kavukcuoglu

Published in NIPS 2016