More importantly, we have a uniﬁed model that can si-APbb
person APmask
person APkp
Faster R-CNN 52.5 - Mask R-CNN, mask-only 53.6 45.8 Mask R-CNN, keypoint-only 50.7 - 64.2
Mask R-CNN, keypoint & mask 52.0 45.1 64.7
Table 5. Multi-task learning of box, mask, and keypoint about the
person category, evaluated on minival . All entries are trained
on the same data for fair comparisons. The backbone is ResNet50-FPN. The entries with 64.2 and 64.7 AP on minival have
test-dev AP of 62.7 and 63.1, respectively (see Table 4).
APkpAPkp
50APkp
75APkp
MAPkp
L
RoIPool 59.8 86.2 66.7 55.1 67.4
RoIAlign 64.2 86.6 69.7 58.7 73.0
Table 6. RoIAlign vs. RoIPool for keypoint detection on
minival . The backbone is ResNet-50-FPN.
multaneously predict boxes, segments, and keypoints while

Mask R-CNN

open source; appendix on more results


minival . The backbone is ResNet-50-FPN.
multaneously predict boxes, segments, and keypoints while
running at 5 fps. Adding a segment branch (for the person category) improves the APkpto 63.1 (Table 4) on
test-dev . More ablations of multi-task learning on
minival are in Table 5. Adding the mask branch to the
box-only ( i.e., Faster R-CNN) or keypoint-only versions
consistently improves these tasks. However, adding the
keypoint branch reduces the box/mask AP slightly, suggesting that while keypoint detection beneﬁts from multitask
training, it does not in turn help the other tasks. Nevertheless, learning all three tasks jointly enables a uniﬁed system
to efﬁciently predict all outputs simultaneously (Figure 7).
We also investigate the effect of RoIAlign on keypoint
detection (Table 6). Though this ResNet-50-FPN backbone
has ﬁner strides ( e.g., 4 pixels on the ﬁnest level), RoIAlign
still shows signiﬁcant improvement over RoIPool and increases APkpby 4.4 points. This is because keypoint detections are more sensitive to localization accuracy. This again

Mask R-CNN

Summary:
We present a conceptually simple, flexible, and general framework for object
instance segmentation. Our approach efficiently detects objects in an image
while simultaneously generating a high-quality segmentation mask for each
instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a
branch for predicting an object mask in parallel with the existing branch for
bounding box recognition. Mask R-CNN is simple to train and adds only a small
overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to
generalize to other tasks, e.g., allowing us to estimate human poses in the
same framework. We show top results in all three tracks of the COCO suite of
challenges, including instance segmentation, bounding-box object detection, and
person keypoint detection. Without bells and whistles, Mask R-CNN outperforms
all existing, single-model entries on every task, including the COCO 2016
challenge winners. We hope our simple and effective approach will serve as a
solid baseline and help ease future research in instance-level recognition.
Code has been made available at: https://github.com/facebookresearch/Detectron

Kaiming He
Georgia Gkioxari
Piotr Dollár
Ross Girshick

open source; appendix on more results


indicates that alignment is essential for pixel-level localization, including masks and keypoints.
Given the effectiveness of Mask R-CNN for extracting
object bounding boxes, masks, and keypoints, we expect it
be an effective framework for other instance-level tasks.
8
training data AP [val] AP AP 50 person rider car truck bus train mcycle bicycle
InstanceCut [23] fine +coarse 15.8 13.0 27.9 10.0 8.0 23.7 14.0 19.5 15.2 9.3 4.7
DWT [4] fine 19.8 15.6 30.0 15.1 11.7 32.9 17.1 20.4 15.0 7.9 4.9
SAIS [17] fine - 17.4 36.7 14.6 12.9 35.7 16.0 23.2 19.0 10.3 7.8
DIN [3] fine +coarse - 20.0 38.8 16.5 16.7 25.7 20.6 30.0 23.4 17.1 10.1
SGN [29] fine +coarse 29.2 25.0 44.9 21.8 20.1 39.4 24.8 33.2 30.8 17.7 12.4

Mask R-CNN

open source; appendix on more results


Mask R-CNN fine 31.5 26.2 49.9 30.5 23.7 46.9 22.8 32.2 18.6 19.1 16.0
Mask R-CNN fine + COCO 36.4 32.0 58.1 34.8 27.0 49.1 30.1 40.9 30.9 24.1 18.7
Table 7. Results on Cityscapes val (‘AP [ val]’ column) and test (remaining columns) sets. Our method uses ResNet-50-FPN.
Appendix A: Experiments on Cityscapes
We further report instance segmentation results on the
Cityscapes [7] dataset. This dataset has fine annotations for 2975 train, 500 val, and 1525 test images. It has
20kcoarse training images without instance annotations,
which we do notuse. All images are 2048 1024 pixels.
The instance segmentation task involves 8 object categories,
whose numbers of instances on the fine training set are:
person rider car truck bus train mcycle bicycle
17.9k 1.8k 26.9k 0.5k 0.4k 0.2k 0.7k 3.7k
Instance segmentation performance on this task is measured

Mask R-CNN

open source; appendix on more results


Instance segmentation performance on this task is measured
by the COCO-style mask AP (averaged over IoU thresholds); AP 50(i.e., mask AP at an IoU of 0.5) is also reported.
Implementation: We apply our Mask R-CNN models with
the ResNet-FPN-50 backbone; we found the 101-layer
counterpart performs similarly due to the small dataset size.
We train with image scale (shorter side) randomly sampled
from [800, 1024], which reduces overﬁtting; inference is on
a single scale of 1024 pixels. We use a mini-batch size of
1 image per GPU (so 8 on 8 GPUs) and train the model
for 24k iterations, starting from a learning rate of 0.01 and
reducing it to 0.001 at 18k iterations. It takes 4 hours of
training on a single 8-GPU machine under this setting.
Results: Table 7 compares our results to the state of the
art on the val andtest sets. Without using the coarse
training set, our method achieves 26.2 AP on test , which
is over 30% relative improvement over the previous best entry (DIN [3]), and is also better than the concurrent work of
SGN’s 25.0 [29]. Both DIN and SGN use fine +coarse

Mask R-CNN

open source; appendix on more results


SGN’s 25.0 [29]. Both DIN and SGN use fine +coarse
data. Compared to the best entry using fine data only
(17.4 AP), we achieve a 50% improvement.
For the person andcarcategories, the Cityscapes dataset
exhibits a large number of within -category overlapping instances (on average 6 people and 9 cars per image). We
argue that within-category overlap is a core difﬁculty of instance segmentation. Our method shows massive improvement on these two categories over the other best entries (relative40% improvement on person from 21.8 to 30.5 and
20% improvement on carfrom 39.4 to 46.9), even though
our method does not exploit the coarse data.
A main challenge of the Cityscapes dataset is training
models in a low-data regime, particularly for the categories
oftruck ,bus, and train , which have about 200-500 traincar:1.00car:0.98
car:0.98 car:0.95 car:0.81car:0.52person:1.00person:1.00person:1.00
person:1.00 person:1.00person:1.00person:1.00
person:1.00 person:1.00person:1.00

Mask R-CNN

open source; appendix on more results


Mask R-CNN

open source; appendix on more results


person:1.00
person:1.00person:1.00
person:1.00
person:1.00person:1.00
person:1.00person:1.00person:1.00
person:1.00person:1.00person:1.00
person:1.00person:1.00
person:1.00person:1.00person:1.00
person:1.00person:1.00
person:1.00person:1.00
person:1.00person:0.99
person:0.99 person:0.98person:0.97
person:0.96person:0.92
person:0.91person:0.70 person:0.59
bicycle:0.99bicycle:0.97car:1.00
car:1.00 car:0.99
car:0.89person:1.00
person:1.00person:1.00
person:1.00
person:1.00person:0.96person:0.93
person:0.89person:0.88person:0.75
rider:0.94
car:1.00
car:1.00car:1.00
car:1.00car:1.00

Mask R-CNN

open source; appendix on more results


rider:0.94
car:1.00
car:1.00car:1.00
car:1.00car:1.00
car:1.00car:1.00 car:0.99car:0.89 car:0.67person:1.00
person:1.00
person:1.00 person:1.00 person:0.82
bus:0.75Figure 8. Mask R-CNN results on Cityscapes test (32.0 AP).
The bottom-right image shows a failure prediction.
ing samples each. To partially remedy this issue, we further
report a result using COCO pre-training. To do this, we initialize the corresponding 7 categories in Cityscapes from a
pre-trained COCO Mask R-CNN model ( rider being randomly initialized). We ﬁne-tune this model for 4k iterations
in which the learning rate is reduced at 3k iterations, which
takes1 hour for training given the COCO model.
The COCO pre-trained Mask R-CNN model achieves
32.0 AP on test , almost a 6 point improvement over the
fine -only counterpart. This indicates the important role
the amount of training data plays. It also suggests that

Mask R-CNN

open source; appendix on more results


32.0 AP on test , almost a 6 point improvement over the
fine -only counterpart. This indicates the important role
the amount of training data plays. It also suggests that
methods on Cityscapes might be inﬂuenced by their lowshot learning performance. We show that using COCO pretraining is an effective strategy on this dataset.
Finally, we observed a bias between the val andtest
AP, as is also observed from the results of [23, 4, 29]. We
found that this bias is mainly caused by the truck ,bus,
and train categories, with the fine -only model having
val/test AP of 28.8/22.8, 53.5/32.2, and 33.0/18.6, respectively. This suggests that there is a domain shift on
these categories, which also have little training data. COCO
pre-training helps to improve results the most on these categories; however, the domain shift persists with 38.0/30.1,
57.5/40.9, and 41.2/30.9 val/test AP, respectively. Note
that for the person andcarcategories we do not see any
such bias ( val/test AP are within1point).
Example results on Cityscapes are shown in Figure 8.
9
description backbone AP AP 50 AP75 APbbAPbb

Mask R-CNN

open source; appendix on more results


+test-time augm. X-152-FPN-NL 41.8 66.0 44.8 47.3 69.3 51.5
Table 8. Enhanced detection results of Mask R-CNN on COCO
minival . Each row adds an extra component to the above row.
We denote ResNeXt model by ‘X’ for notational brevity.
Appendix B: Enhanced Results on COCO
As a general framework, Mask R-CNN is compatible with complementary techniques developed for detection/segmentation, including improvements made to
Fast/Faster R-CNN and FCNs. In this appendix we describe some techniques that improve over our original results. Thanks to its generality and ﬂexibility, Mask R-CNN
was used as the framework by the three winning teams in
the COCO 2017 instance segmentation competition, which
all signiﬁcantly outperformed the previous state of the art.
Instance Segmentation and Object Detection
We report some enhanced results of Mask R-CNN in Table 8. Overall, the improvements increase mask AP 5.1
points (from 36.7 to 41.8) and box AP 7.7 points (from 39.6
to 47.3). Each model improvement increases both mask AP
and box AP consistently, showing good generalization of
the Mask R-CNN framework. We detail the improvements

Mask R-CNN

open source; appendix on more results


to 47.3). Each model improvement increases both mask AP
and box AP consistently, showing good generalization of
the Mask R-CNN framework. We detail the improvements
next. These results, along with future updates, can be reproduced by our released code at https://github.com/
facebookresearch/Detectron , and can serve as
higher baselines for future research.
Updated baseline: We start with an updated baseline
with a different set of hyper-parameters. We lengthen the
training to 180k iterations, in which the learning rate is reduced by 10 at 120k and 160k iterations. We also change
the NMS threshold to 0.5 (from a default value of 0.3). The
updated baseline has 37.0 mask AP and 40.5 box AP.
End-to-end training: All previous results used stagewise training, i.e., training RPN as the ﬁrst stage and Mask
R-CNN as the second. Following [37], we evaluate endto-end (‘e2e’) training that jointly trains RPN and Mask RCNN. We adopt the ‘approximate’ version in [37] that only
computes partial gradients in the RoIAlign layer by ignoring the gradient w.r.t. RoI coordinates. Table 8 shows that

Mask R-CNN

open source; appendix on more results


computes partial gradients in the RoIAlign layer by ignoring the gradient w.r.t. RoI coordinates. Table 8 shows that
e2e training improves mask AP by 0.6 and box AP by 1.2.
ImageNet-5k pre-training: Following [45], we experiment with models pre-trained on a 5k-class subset of ImageNet (in contrast to the standard 1k-class subset). This 5
increase in pre-training data improves both mask and box 1
AP. As a reference, [40] used 250more images (300M)
and reported a 2-3 box AP improvement on their baselines.description backbone APkpAPkp
50APkp
75APkp
MAPkp
L
original baseline R-50-FPN 64.2 86.6 69.7 58.7 73.0
+updated baseline R-50-FPN 65.1 86.6 70.9 59.9 73.6
+deeper R-101-FPN 66.1 87.7 71.7 60.5 75.0
+ResNeXt X-101-FPN 67.3 88.0 73.3 62.2 75.6

Mask R-CNN

open source; appendix on more results


+ResNeXt X-101-FPN 67.3 88.0 73.3 62.2 75.6
+data distillation [35] X-101-FPN 69.1 88.9 75.3 64.1 77.1
+test-time augm. X-101-FPN 70.4 89.3 76.8 65.8 78.1
Table 9. Enhanced keypoint results of Mask R-CNN on COCO
minival . Each row adds an extra component to the above row.
Here we use only keypoint annotations but no mask annotations.
We denote ResNet by ‘R’ and ResNeXt by ‘X’ for brevity.
Train-time augmentation: Scale augmentation at train
time further improves results. During training, we randomly
sample a scale from [640, 800] pixels and we increase the
number of iterations to 260k (with the learning rate reduced
by 10 at 200k and 240k iterations). Train-time augmentation improves mask AP by 0.6 and box AP by 0.8.
Model architecture: By upgrading the 101-layer
ResNeXt to its 152-layer counterpart [19], we observe an
increase of 0.5 mask AP and 0.6 box AP. This shows a
deeper model can still improve results on COCO.

Mask R-CNN

open source; appendix on more results


increase of 0.5 mask AP and 0.6 box AP. This shows a
deeper model can still improve results on COCO.
Using the recently proposed non-local (NL) model [43],
we achieve 40.3 mask AP and 45.0 box AP. This result is
without test-time augmentation, and the method runs at 3fps
on an Nvidia Tesla P100 GPU at test time.
Test-time augmentation: We combine the model results
evaluated using scales of [400, 1200] pixels with a step of
100 and on their horizontal ﬂips. This gives us a singlemodel result of 41.8 mask AP and 47.3 box AP.
The above result is the foundation of our submission to
the COCO 2017 competition (which also used an ensemble,
not discussed here). The ﬁrst three winning teams for the
instance segmentation task were all reportedly based on an
extension of the Mask R-CNN framework.
Keypoint Detection
We report enhanced results of keypoint detection in Table 9. As an updated baseline, we extend the training schedule to 130k iterations in which the learning rate is reduced
by 10 at 100k and 120k iterations. This improves APkpby
about 1 point. Replacing ResNet-50 with ResNet-101 and

Mask R-CNN

open source; appendix on more results


by 10 at 100k and 120k iterations. This improves APkpby
about 1 point. Replacing ResNet-50 with ResNet-101 and
ResNeXt-101 increases APkpto 66.1 and 67.3, respectively.
With a recent method called data distillation [35], we are
able to exploit the additional 120k unlabeled images provided by COCO. In brief, data distillation is a self-training
strategy that uses a model trained on labeled data to predict annotations on unlabeled images, and in turn updates
the model with these new annotations. Mask R-CNN provides an effective framework for such a self-training strategy. With data distillation, Mask R-CNN APkpimprove by
1.8 points to 69.1. We observe that Mask R-CNN can beneﬁt from extra data, even if that data is unlabeled .
By using the same test-time augmentation as used for
instance segmentation, we further boost APkpto 70.4.
10
Acknowledgements: We would like to acknowledge Ilija
Radosavovic for contributions to code release and enhanced
results, and the Caffe2 team for engineering support.
References
[1] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2D

Mask R-CNN

open source; appendix on more results


References
[1] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2D
human pose estimation: New benchmark and state of the art
analysis. In CVPR , 2014. 8
[2] P. Arbel ´aez, J. Pont-Tuset, J. T. Barron, F. Marques, and
J. Malik. Multiscale combinatorial grouping. In CVPR ,
2014. 2
[3] A. Arnab and P. H. Torr. Pixelwise instance segmentation
with a dynamically instantiated network. In CVPR , 2017. 3,
9
[4] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In CVPR , 2017. 3, 9
[5] S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Insideoutside net: Detecting objects in context with skip pooling
and recurrent neural networks. In CVPR , 2016. 5
[6] Z. Cao, T. Simon, S.-E. Wei, and Y . Sheikh. Realtime multiperson 2d pose estimation using part afﬁnity ﬁelds. In CVPR ,
2017. 7, 8

Mask R-CNN

open source; appendix on more results


2017. 7, 8
[7] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
Cityscapes dataset for semantic urban scene understanding.
InCVPR , 2016. 9
[8] J. Dai, K. He, Y . Li, S. Ren, and J. Sun. Instance-sensitive
fully convolutional networks. In ECCV , 2016. 2
[9] J. Dai, K. He, and J. Sun. Convolutional feature masking for
joint object and stuff segmentation. In CVPR , 2015. 2
[10] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR , 2016. 2, 3,
4, 5, 6
[11] J. Dai, Y . Li, K. He, and J. Sun. R-FCN: Object detection via
region-based fully convolutional networks. In NIPS , 2016. 2
[12] R. Girshick. Fast R-CNN. In ICCV , 2015. 1, 2, 3, 4, 6

Mask R-CNN

open source; appendix on more results


[12] R. Girshick. Fast R-CNN. In ICCV , 2015. 1, 2, 3, 4, 6
[13] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014. 2, 3
[14] R. Girshick, F. Iandola, T. Darrell, and J. Malik. Deformable
part models are convolutional neural networks. In CVPR ,
2015. 4
[15] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV . 2014. 2
[16] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and ﬁne-grained localization. In CVPR , 2015. 2
[17] Z. Hayder, X. He, and M. Salzmann. Shape-aware instance
segmentation. In CVPR , 2017. 9
[18] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling
in deep convolutional networks for visual recognition. In

Mask R-CNN

open source; appendix on more results


[18] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling
in deep convolutional networks for visual recognition. In
ECCV . 2014. 1, 2
[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR , 2016. 2, 4, 7, 10
[20] J. Hosang, R. Benenson, P. Doll ´ar, and B. Schiele. What
makes for effective detection proposals? PAMI , 2015. 2[21] J. Huang, V . Rathod, C. Sun, M. Zhu, A. Korattikara,
A. Fathi, I. Fischer, Z. Wojna, Y . Song, S. Guadarrama, et al.
Speed/accuracy trade-offs for modern convolutional object
detectors. In CVPR , 2017. 2, 3, 4, 6, 7
[22] M. Jaderberg, K. Simonyan, A. Zisserman, and
K. Kavukcuoglu. Spatial transformer networks. In
NIPS , 2015. 4
[23] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and

Mask R-CNN

open source; appendix on more results


NIPS , 2015. 4
[23] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and
C. Rother. Instancecut: from edges to instances with multicut. In CVPR , 2017. 3, 9
[24] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classiﬁcation with deep convolutional neural networks. In NIPS ,
2012. 2
[25] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel. Backpropagation
applied to handwritten zip code recognition. Neural computation , 1989. 2
[26] Y . Li, H. Qi, J. Dai, X. Ji, and Y . Wei. Fully convolutional
instance-aware semantic segmentation. In CVPR , 2017. 2,
3, 5, 6
[27] T.-Y . Lin, P. Doll ´ar, R. Girshick, K. He, B. Hariharan, and
S. Belongie. Feature pyramid networks for object detection.
InCVPR , 2017. 2, 4, 5, 7

Mask R-CNN

open source; appendix on more results


S. Belongie. Feature pyramid networks for object detection.
InCVPR , 2017. 2, 4, 5, 7
[28] T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV , 2014. 2, 5
[29] S. Liu, J. Jia, S. Fidler, and R. Urtasun. SGN: Sequential grouping networks for instance segmentation. In ICCV ,
2017. 3, 9
[30] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In CVPR , 2015. 1, 3, 6
[31] V . Nair and G. E. Hinton. Rectiﬁed linear units improve restricted boltzmann machines. In ICML , 2010. 4
[32] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy. Towards accurate multiperson pose estimation in the wild. In CVPR , 2017. 8

Mask R-CNN

open source; appendix on more results


[33] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In NIPS , 2015. 2, 3
[34] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Doll ´ar. Learning to reﬁne object segments. In ECCV , 2016. 2, 3
[35] I. Radosavovic, P. Doll ´ar, R. Girshick, G. Gkioxari, and
K. He. Data distillation: Towards omni-supervised learning.
arXiv:1712.04440 , 2017. 10
[36] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS , 2015. 1, 2, 3, 4, 7
[37] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In TPAMI , 2017. 10
[38] A. Shrivastava, A. Gupta, and R. Girshick. Training regionbased object detectors with online hard example mining. In
CVPR , 2016. 2, 5

Mask R-CNN

open source; appendix on more results


CVPR , 2016. 2, 5
[39] A. Shrivastava, R. Sukthankar, J. Malik, and A. Gupta. Beyond skip connections: Top-down modulation for object detection. arXiv:1612.06851 , 2016. 4, 7
[40] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting
unreasonable effectiveness of data in deep learning era. In
ICCV , 2017. 10
11
[41] C. Szegedy, S. Ioffe, and V . Vanhoucke. Inception-v4,
inception-resnet and the impact of residual connections on
learning. In ICLR Workshop , 2016. 7
[42] J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W.
Smeulders. Selective search for object recognition. IJCV ,
2013. 2
[43] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural
networks. arXiv:1711.07971 , 2017. 10

Mask R-CNN

open source; appendix on more results


networks. arXiv:1711.07971 , 2017. 10
[44] S.-E. Wei, V . Ramakrishna, T. Kanade, and Y . Sheikh. Convolutional pose machines. In CVPR , 2016. 8
[45] S. Xie, R. Girshick, P. Doll ´ar, Z. Tu, and K. He. Aggregated
residual transformations for deep neural networks. In CVPR ,
2017. 4, 10
12

Mask R-CNN

open source; appendix on more results


Pixelwise Instance Segmentation with a Dynamically Instantiated Network
Anurag Arnab and Philip H.S Torr
University of Oxford
{anurag.arnab, philip.torr }@eng.ox.ac.uk
Abstract
Semantic segmentation and object detection research
have recently achieved rapid progress. However, the former
task has no notion of different instances of the same object,
and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a
segmentation map where each pixel is assigned an object
class and instance identity label. Most approaches adapt
object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic segmentation module, which feeds into an instance subnetwork.
This subnetwork uses the initial category-level segmentation, along with cues from the output of an object detector,
within an end-to-end CRF to predict instances. This part
of our model is dynamically instantiated to produce a variable number of instances per image. Our end-to-end approach requires no post-processing and considers the image holistically, instead of processing independent proposals. Therefore, unlike some related work, a pixel cannot belong to multiple instances. Furthermore, far more precise
segmentations are achieved, as shown by our substantial

Summary:
Semantic segmentation and object detection research have recently achieved
rapid progress. However, the former task has no notion of different instances
of the same object, and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a segmentation map where
each pixel is assigned an object class and instance identity label. Most
approaches adapt object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic segmentation module, which
feeds into an instance subnetwork. This subnetwork uses the initial
category-level segmentation, along with cues from the output of an object
detector, within an end-to-end CRF to predict instances. This part of our model
is dynamically instantiated to produce a variable number of instances per
image. Our end-to-end approach requires no post-processing and considers the
image holistically, instead of processing independent proposals. Therefore,
unlike some related work, a pixel cannot belong to multiple instances.
Furthermore, far more precise segmentations are achieved, as shown by our
state-of-the-art results (particularly at high IoU thresholds) on the Pascal
VOC and Cityscapes datasets.

CVPR 2017

1. Multiscale Combinatorial Grouping

P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, J. Malik

2. Higher Order Conditional Random Fields in Deep Neural Networks

A. Arnab, S. Jayasumana, S. Zheng, P. H. S. Torr

3. Bottom-up Instance Segmentation with Deep Higher Order CRFs

4. Deep Watershed Transform for Instance Segmentation

5. Learning Deep Structured Models

L. Chen, A. Schwing, A. Yuille, R. Urtasun

6. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille


segmentations are achieved, as shown by our substantial
improvements at high APrthresholds.
1. Introduction
Semantic segmentation and object detection are wellstudied scene understanding problems, and have recently
witnessed great progress due to deep learning [ 22,13,7].
However, semantic segmentation – which labels every pixel
in an image with its object class – has no notion of different
instances of an object (Fig. 1). Object detection does localise different object instances, but does so at a very coarse,
bounding-box level. Instance segmentation localises objects at a pixel level, as shown in Fig. 1, and can be thought
of being at the intersection of these two scene understanding
tasks. Unlike the former, it knows about different instances
of the same object, and unlike the latter, it operates at a pixel
level. Accurate recognition and localisation of objects enables many applications, such as autonomous driving [ 9],
image-editing [ 53] and robotics [ 17].
Many recent approaches to instance segmentation arebased on object detection pipelines where objects are ﬁrst
localised with bounding boxes. Thereafter, each bounding

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


Many recent approaches to instance segmentation arebased on object detection pipelines where objects are ﬁrst
localised with bounding boxes. Thereafter, each bounding
box is reﬁned into a segmentation [ 19,20,32,37,30]. Another related approach [ 12,56] is to use segment-based region proposals [ 10,41,42] instead of box-based proposals.
However, these methods do not consider the entire image,
but rather independent proposals. As a result, occlusions
between different objects are not handled. Furthermore,
many of these methods cannot easily produce segmentation
maps of the image, as shown in Fig. 1, since they process
numerous proposals independently. There are typically far
more proposals than actual objects in the image, and these
proposals can overlap and be assigned different class labels.
Finally, as these methods are based on an initial detection
step, they cannot recover from false detections.
Our proposed method is inspired by the fact that instance
segmentation can be viewed as a more complex form of semantic segmentation, since we are not only required to label the object class of each pixel, but also its instance identity. We produce a pixelwise segmentation of the image,

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


where each pixel is assigned both a semantic class and instance label. Our end-to-end trained network, which outputs a variable number of instances per input image, begins with an initial semantic segmentation module. The
following, dynamic part of the network, then uses information from an object detector and a Conditional Random
Field (CRF) model to distinguish different instances. This
approach is robust to false-positive detections, as well as
poorly localised bounding boxes which do not cover the
entire object, in contrast to detection-based methods to instance segmentation. Moreover, as it considers the entire
image when making predictions, it attempts to resolve occlusions between different objects and can produce segmentation maps as in Fig. 1without any post-processing.
Furthermore, we note that the Average Precision (AP)
metric [ 14] used in evaluating object detection systems, and
itsAPrvariant [ 19] used for instance segmentation, considers individual, potentially overlapping, object predictions in
isolation, as opposed to the entire image. To evaluate methods such as ours, which produce complete segmentation
maps and reason about occlusions, we also evaluate using
1
arXiv:1704.02386v1  [cs.CV]  7 Apr 2017
(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation
Figure 1: Object detection (a) localises the different people, but at a coarse, bounding-box level. Semantic segmentation
(b) labels every pixel, but has no notion of instances. Instance segmentation (c) labels each pixel of each person uniquely.
Our proposed method jointly produces both semantic and instance segmentations. Our method uses the output of an object
detector as a cue to identify instances, but is robust to false positive detections, poor bounding box localisation and occlusions.
Best viewed in colour.
the “Matching Intersection over Union” metric.
Our system, which is based on an initial semantic segmentation subnetwork, produces sharp and accurate instance segmentations. This is reﬂected by the substantial improvements we achieve over state-of-the-art methods
at highAPrthresholds on the Pascal VOC and Semantic
Boundaries datasets. Furthermore, our network improves
on the semantic segmentation task while being trained for
the related task of instance segmentation.
2. Related Work
An early work on instance segmentation was by Winn
and Shotton [ 51]. A per-pixel unary classiﬁer was trained

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


2. Related Work
An early work on instance segmentation was by Winn
and Shotton [ 51]. A per-pixel unary classiﬁer was trained
to predict parts of an object. These parts were then encouraged to maintain a spatial ordering, that is characteristic of
an instance, using asymmetric pairwise potentials in a Conditional Random Field (CRF). Subsequent work [ 54], presented another approach where detection outputs of DPM
[15], with associated foreground masks, were assigned a
depth ordering using a generative, probabilistic model. This
depth ordering resolved occlusions.
However, instance segmentation has become more common after the “Simultaneous Detection and Segmentation”
(SDS) work of Hariharan et al. [19]. This system was based
on the R-CNN pipeline [ 16]: Region proposals, generated
by the method of [ 1], were classiﬁed into object categories
with a Convolutional Neural Network (CNN) before applying bounding-box regression as post-processing. A classspeciﬁc segmentation was then performed in this bounding

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


with a Convolutional Neural Network (CNN) before applying bounding-box regression as post-processing. A classspeciﬁc segmentation was then performed in this bounding
box to simultaneously detect and segment the object. Numerous works [ 20,8,30] have extended this pipeline. However, approaches that segment instances by reﬁning detections [ 19,20,8,11,30] are inherently limited by the quality of the initial proposals. This problem is exacerbated by
the fact that this pipeline consists of several different modules trained with different objective functions. Furthermore,numerous post-processing steps such as “superpixel projection” and rescoring are performed. Dai et al. [12] addressed
some of these issues by designing one end-to-end trained
network that generates box-proposals, creates foreground
masks from these proposals and then classiﬁes these masks.
This network can be seen as an extension of the end-to-end
Faster-RCNN [ 44] detection framework, which generates
box-proposals and classiﬁes them. Additionally, Liu et al.
[37] formulated an end-to-end version of the SDS network
[19], whilst [ 32] iteratively reﬁned object proposals.
On a separate track, algorithms have also been developed

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


[19], whilst [ 32] iteratively reﬁned object proposals.
On a separate track, algorithms have also been developed
that do not require object detectors. Zhang et al. [57,58]
segmented car instances by predicting the depth ordering
of each pixel in the image. Unlike the previous detectionbased approaches, this method reasoned globally about all
instances in the image simultaneously (rather than individual proposals) with an MRF-based formulation. However,
inference of this graphical model was not performed end-toend as shown to be possible in [ 60,2,5,34]. Furthermore,
although this method does not use object detections, it is
trained with ground truth depth and assumes a maximum
of nine cars in an image. Predicting all the instances in
an image simultaneously (rather than classifying individual
proposals) requires a model to be able to handle a variable
number of output instances per image. As a result, [ 45]
proposed a Recurrent Neural Network (RNN) for this task.
However, this model was only for a single object category.
Our proposed method not only outputs a variable number of
instances, but can also handle multiple object classes.
Liang et al. [33] developed another proposal-free method
based on the semantic segmentation network of [ 6]. The

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


instances, but can also handle multiple object classes.
Liang et al. [33] developed another proposal-free method
based on the semantic segmentation network of [ 6]. The
category-level segmentation, along with CNN features, was
used to predict instance-level bounding boxes. The number
of instances of each class was also predicted to enable a ﬁnal
spectral clustering step. However, this additional information predicted by Liang’s network could have been obtained
 Semantic Segmentation subnetwork  Instance Segmentation subnetwork Instance Unary Potentials Pixelwise
CNN Semantic CRF Detector Box
Global
Shape Instance CRF
Input Image Instance Segmentation
Semantic Segmentation End-to-end Network
K+1
D+1
Figure 2: Network overview: Our end-to-end trained network consists of semantic- and instance-segmentation modules. The
intermediate category-level segmentation, along with the outputs of an object detector, are used to reason about instances.
This is done by instance unary terms which use information from the detector’s bounding boxes, the initial semantic segmentation and also the object’s shape. A ﬁnal CRF is used to combine all this information together to obtain an instance

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


segmentation. The output of the semantic segmentation module is a ﬁxed size W×H×(K+ 1) tensor where Kis the
number of object classes, excluding background, in the dataset. The ﬁnal output, however, is of a variable W×H×(D+ 1)
dimensions where Dis the number of detected objects (and one background label).
from an object detector. Arnab et al. [3] also started with an
initial semantic segmentation network [ 2], and combined
this with the outputs of an object detector using a CRF to
reason about instances. This method was not trained endto-end though, and could not really recover from errors in
bounding-box localisation or occlusion.
Our method also has an initial semantic segmentation
subnetwork, and uses the outputs of an object detector.
However, in contrast to [ 3] it is trained end-to-end to improve on both semantic- and instance-segmentation performance (to our knowledge, this is the ﬁrst work to achieve
this). Furthermore, it can handle detector localisation errors and occlusions better due to the energy terms in our
end-to-end CRF. In contrast to detection-based approaches
[19,20,12,37], our network requires no additional postprocessing to create an instance segmentation map as in

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


[19,20,12,37], our network requires no additional postprocessing to create an instance segmentation map as in
Fig. 1(c) and reasons about the entire image, rather than
independent proposals. This global reasoning allows our
method to produce more accurate segmentations. Our proposed system also handles a variable number of instances
per image, and thus does not assume a maximum number
of instances like [ 57,58].
3. Proposed Approach
Our network (Fig. 2) contains an initial semantic segmentation module. We use the semantic segmentation result, along with the outputs of an object detector, to compute
the unary potentials of a Conditional Random Field (CRF)
deﬁned over object instances. We perform mean ﬁeld inference in this random ﬁeld to obtain the Maximum a Posteriori (MAP) estimate, which is our labelling. Although ournetwork consists of two conceptually different parts – a semantic segmentation module, and an instance segmentation
network – the entire pipeline is fully differentiable, given
object detections, and trained end-to-end.
3.1. Semantic Segmentation subnetwork
Semantic Segmentation assigns each pixel in an image a
semantic class label from a given set, L. In our case, this
module uses the FCN8s architecture [ 38] which is based

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


semantic class label from a given set, L. In our case, this
module uses the FCN8s architecture [ 38] which is based
on the VGG [ 47] ImageNet model. For better segmentation results, we include mean ﬁeld inference of a Conditional Random Field as the last layer of this module. This
CRF contains the densely-connected pairwise potentials described in [ 26] and is formulated as a recurrent neural network as in [ 60]. Additionally, we include the Higher Order
detection potential described in [ 2]. This detection potential has two primary beneﬁts: Firstly, it improves semantic
segmentation quality by encouraging consistency between
object detections and segmentations. Secondly, it also recalibrates detection scores. This detection potential is similar to the one previously proposed by [ 28], [48], [52] and
[55], but formulated for the differentiable mean ﬁeld inference algorithm. We employ this potential as we are already
using object detection information for identifying object instances in the next stage. We denote the output at the semantic segmentation module of our network as the tensor
Q, whereQi(l)denotes the probability (obtained by applying the softmax function on the network’s activations) of
pixelitaking on the label l∈ L.

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


pixelitaking on the label l∈ L.
(a) Semantic Segmentation (b) Instance Segmentation
Figure 3: Instance segmentation using only the “Box” unary
potential. This potential is effective when we have a good
initial semantic segmentation (a). Occlusions between objects of the same class can be resolved by the pairwise term
based on appearance differences. Note that we can ignore
the conﬁdent, false-positive “bottle” detections (b). This is
in contrast to methods such as [ 8,19,20,30] which cannot
recover from detection errors.
3.2. Instance Segmentation subnetwork
At the input to our instance segmentation subnetwork,
we assume that we have two inputs available: The semantic
segmentation predictions, Q, for each pixel and label, and a
set of object detections. For each input image, we assume
that there are Dobject detections, and that the ithdetection
is of the form (li,si,Bi)whereli∈ Lis the detected class
label,si∈[0,1]is the conﬁdence score and Biis the set of
indices of the pixels falling within the detector’s bounding

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


label,si∈[0,1]is the conﬁdence score and Biis the set of
indices of the pixels falling within the detector’s bounding
box. Note that the number Dvaries for every input image.
The problem of instance segmentation can then be
thought of as assigning every pixel to either a particular object detection, or the background label. This is based on
the assumption that every object detection speciﬁes a potential object instance. We deﬁne a multinomial random
variable,V, at each of the Npixels in the image, and
V= [V1V2...V N]T. Each variable at pixel i,Vi, is assigned a label corresponding to its instance. This label set,
{0,1,2,...,D }changes for each image since D, the number
of detections, varies for every image (0 is the background
label). In the case of instance segmentation of images, the
quality of a prediction is invariant to the permutations of the
instance labelling. For example, labelling the “blue person”
in Fig. 1(c) as “1” and the “purple person” as “2” is no different to labelling them as “2” and “1” respectively. This

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


condition is handled by our loss function in Sec. 3.4.
Note that unlike works such as [ 57] and [ 58] we do not
assume a maximum number of possible instances and keep
a ﬁxed label set. Furthermore, since we are considering object detection outputs jointly with semantic segmentation
predictions, we have some robustness to high-scoring false
positive detections unlike methods such as [ 8,20,37] which
reﬁne object detections into segmentations.
We formulate a Conditional Random Field over our instance variables, V, which consists of unary and pairwise
(a) Only Box term (b) Box and Global terms
Figure 4: The “Global” unary potential (b) is particularly
effective in cases where the input detection bounding box
does not cover the entire extent of the object. Methods
which are based on reﬁning bounding-box detections such
as [19,20,8,12] cannot cope with poorly localised detections. Note, the overlaid detection boxes are an additional
input to our system.
energies. The energy of the assignment vto all the variables,V, is
E(V=v) =/summationdisplay
iU(vi) +/summationdisplay
i<jP(vi,vj). (1)

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


E(V=v) =/summationdisplay
iU(vi) +/summationdisplay
i<jP(vi,vj). (1)
The unary energy is a sum of three terms, which take into
account the object detection bounding boxes, the initial semantic segmentation and shape information,
U(vi) =−ln[w1ψBox(vi) +w2ψGlobal (vi)+
w3ψShape (vi)], (2)
and are described further in Sections 3.2.1 through 3.2.3 .
w1,w2andw3are all weighting co-efﬁcients learned via
backpropagation.
3.2.1 Box Term
This potential encourages a pixel to be assigned to the instance corresponding to the kthdetection if it falls within
the detection’s bounding box. This potential is proportional
to the probability of the pixel’s semantic class being equal
to the detected class Qi(lk)and the detection score, sk.
ψBox(Vi=k) =/braceleftBigg
Qi(lk)skifi∈Bk
0 otherwise(3)
As shown in Fig. 3, this potential performs well when the
initial semantic segmentation is good. It is robust to false

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


0 otherwise(3)
As shown in Fig. 3, this potential performs well when the
initial semantic segmentation is good. It is robust to false
positive detections, unlike methods which reﬁne bounding
boxes [ 8,19,20] since the detections are considered in light
of our initial semantic segmentation, Q. Together with the
pairwise term (Sec. 3.2.4 ), occlusions between objects of
the same class can be resolved if there are appearance differences in the different instances.
3.2.2 Global Term
This term does not rely on bounding boxes, but only the segmentation prediction at a particular pixel, Qi. It encodes the
intuition that if we only know there are dpossible instances
of a particular object class, and have no further localisation information, each instance is equally probable, and this
potential is proportional to the semantic segmentation conﬁdence for the detected object class at that pixel:
ψGlobal (Vi=k) =Qi(lk). (4)
As shown in Fig. 4, this potential overcomes cases where
the bounding box does not cover the entire extent of the object, as it assigns probability mass to a particular instance

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


As shown in Fig. 4, this potential overcomes cases where
the bounding box does not cover the entire extent of the object, as it assigns probability mass to a particular instance
label throughout all pixels in the image. This is also beneﬁcial during training, as it ensures that the ﬁnal output is
dependent on the segmentation prediction at all pixels in the
image, leading to error gradients that are more stable across
batches and thus more amenable to backpropagation.
3.2.3 Shape Term
We also incorporate shape priors to help us reason about occlusions involving multiple objects of the same class, which
may have minimal appearance variation between them, as
shown in Fig. 5. In such cases, a prior on the expected shape
of an object category can help us to identify the foreground
instance within a bounding box. Previous approaches to incorporating shape priors in segmentation [ 23,8,50] have
involved generating “shape exemplars” from the training
dataset and, at inference time, matching these exemplars to
object proposals using the Chamfer distance [ 46,36].
We propose a fully differentiable method: Given a set of
shape templates, T, we warp each shape template using bilinear interpolation into ˜Tso that it matches the dimensions

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


We propose a fully differentiable method: Given a set of
shape templates, T, we warp each shape template using bilinear interpolation into ˜Tso that it matches the dimensions
of thekthbounding box, Bk. We then select the shape prior
which matches the segmentation prediction for the detected
class within the bounding box, QBk(lk), the best according to the normalised cross correlation. Our shape prior is
then the Hadamard (elementwise) product ( ⊙) between the
segmentation unaries and the matched shape prior:
t∗= arg max
t∈˜T/summationtextQBk(lk)⊙t/vextenddouble/vextenddoubleQBk(lk)/vextenddouble/vextenddouble/bardblt/bardbl(5)
ψ(VBk=k) =QBk(lk)⊙t∗. (6)
Equations 5and6can be seen as a special case of maxpooling, and the numerator of Eq. 5is simply a convolution
that produces a scalar output since the two arguments are of
equal dimension. Additionally, during training, we can consider the shape priors Tas parameters of our “shape term”
layer and backpropagate through to the matched exemplar

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


equal dimension. Additionally, during training, we can consider the shape priors Tas parameters of our “shape term”
layer and backpropagate through to the matched exemplar
t∗to update it. In practice, we initialised these parameters
(a) Without shape term (b) With Shape term
Figure 5: The “Shape” unary potential (b) helps us to distinguish between the green and purple sheep, which the other
two unary potentials cannot. Input detections are overlaid
on the images.
with the shape priors described in [ 50]. This consists of
roughly 250 shape templates for each of ﬁve different aspect ratios. These were obtained by clustering foreground
masks of object instances from the training set.
Here, we have only matched a single shape template to
a proposed instance. This method could be extended in future to matching multiple templates to an instance, in which
case each shape exemplar would correspond to a part of the
object such as in DPM [ 15].
3.2.4 Pairwise term
The pairwise term consists of densely-connected Gaussian
potentials [ 26] and encourages appearance and spatial consistency. The weights governing the importance of these
terms are also learnt via backpropagation, as in [ 60]. We

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


potentials [ 26] and encourages appearance and spatial consistency. The weights governing the importance of these
terms are also learnt via backpropagation, as in [ 60]. We
ﬁnd that these priors are useful in the case of instance segmentation as well, since nearby pixels that have similar appearance often belong to the same object instance. They are
often able to resolve occlusions based on appearance differences between objects of the same class (Fig. 3).
3.3. Inference of our Dynamic Instance CRF
We use mean ﬁeld inference to approximately minimise
the Gibbs Energy in Eq. 1which corresponds to ﬁnding the
Maximum a Posteriori (MAP) labelling of the corresponding probability distribution, P(V=v) =1
Zexp (−E(v))
whereZis the normalisation factor. Mean ﬁeld inference is
differentiable, and this iterative algorithm can be unrolled
and seen as a recurrent neural network [ 60]. Following this
approach, we can incorporate mean ﬁeld inference of a CRF
as a layer of our neural network. This enables us to train our
entire instance segmentation network end-to-end.
Because we deal with a variable number of instances for
every image, our CRF needs to be dynamically instantiated

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


entire instance segmentation network end-to-end.
Because we deal with a variable number of instances for
every image, our CRF needs to be dynamically instantiated
to have a different number of labels for every image, as observed in [ 3]. Therefore, unlike [ 60], none of our weights
are class-speciﬁc. This weight-sharing not only allows us to
deal with variable length inputs, but class-speciﬁc weights
(a) Original
ground truth, G(b) Prediction, P (c) “Matched”
ground truth, G∗
Figure 6: Due to the problem of label permutations, we
“match” the ground truth with our prediction before computing the loss when training.
also do not make sense in the case of instance segmentation
since a class label has no particular semantic meaning.
3.4. Loss Function
When training for instance segmentation, we have a
single loss function which we backpropagate through our
instance- and semantic-segmentation modules to update all
the parameters. As discussed previously, we need to deal
with different permutations of our ﬁnal labelling which
could have the same ﬁnal result. The works of [ 57] and [ 58]
order instances by depth to break this symmetry. However,

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


could have the same ﬁnal result. The works of [ 57] and [ 58]
order instances by depth to break this symmetry. However,
this requires ground-truth depth maps during training which
we do not assume that we have. Proposal-based methods
[12,19,20,37] do not have this issue since they consider a
single proposal at a time, rather than the entire image. Our
approach is similar to [ 45] in that we match the original
ground truth to our instance segmentation prediction based
on the Intersection over Union (IoU) [ 14] of each instance
prediction and ground truth, as shown in Fig. 6.
More formally, we denote the ground-truth labelling of
an image, G, to be a set of rsegments, {g1,g2,...,g r},
where each segment (set of pixels) is an object instance
and has an associated semantic class label. Our prediction, which is the output of our network, P, is a set of s
segments, {p1,p2,...,p s}, also where each segment corresponds to an instance label and also has an associated class
label. Note that randsmay be different since we may predict greater or fewer instances than actually present. Let
Mdenote the set of all permutations of the ground-truth,

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


label. Note that randsmay be different since we may predict greater or fewer instances than actually present. Let
Mdenote the set of all permutations of the ground-truth,
G. As can be seen in Fig. 6, different permutations of the
ground-truth correspond to the same qualitative result. We
deﬁne the “matched” ground-truth, G∗, as the permutation
of the original ground-truth labelling which maximises the
IoU between the prediction, P, and ground truth:
G∗= arg max
m∈MIoU(m,P). (7)
Once we have the “matched” ground truth, G∗, (Fig. 6) foran image, we can apply any loss function to train our network for segmentation. In our case, we use the common
cross-entropy loss function. We found that this performed
better than the approximate IoU loss proposed in [ 27,45].
Crucially, we do not need to evaluate all permutations
of the ground truth to compute Eq. 7, since it can be formulated as a maximum-weight bipartite matching problem.
The edges in our bipartite graph connect ground-truth and
predicted segments. The edge weights are given by the
IoU between the ground truth and predicted segments if

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


The edges in our bipartite graph connect ground-truth and
predicted segments. The edge weights are given by the
IoU between the ground truth and predicted segments if
they share the same semantic class label, and zero otherwise. Leftover segments are matched to “dummy” nodes
with zero overlap.
Additionally, the ordering of the instances in our network
are actually determined by the object detector, which remains static during training. As a result, the ordering of
our predictions does not ﬂuctuate much during training – it
only changes in cases where there are multiple detections
overlapping an object.
3.5. Network Training
We ﬁrst train a network for semantic segmentation with
the standard cross-entropy loss. In our case, this network is
FCN8s [ 38] with a CRF whose inference is unrolled as an
RNN and trained end-to-end, as described in [ 60] and [ 2].
To this pretrained network, we append our instance segmentation subnetwork, and ﬁnetune with instance segmentation
annotations and only the loss detailed in Sec. 3.4. For the
semantic segmentation subnetwork, we train with an initial
learning rate of 10−8, momentum of 0.9 and batch size of
20. The learning rate is low since we do not normalise the

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


learning rate of 10−8, momentum of 0.9 and batch size of
20. The learning rate is low since we do not normalise the
loss by the number of pixels. This is so that images with
more pixels contribute a higher loss. The normalised learning rate is approximately 2×10−3. When training our instance segmentation network as well, we lower the learning
rate to 10−12and use a batch size of 1 instead. Decreasing the batch size gave empirically better results. We also
clipped gradients (a technique common in training RNNs
[40]) withℓ2norms above 109. This threshold was set by
observing “normal” gradient magnitudes during training.
The relatively high magnitude is due to the fact that our
loss is not normalised. In our complete network, we have
two CRF inference modules which are RNNs (one each in
the semantic- and instance-segmentation subnetworks), and
gradient clipping facilitated successful training.
3.6. Discussion
Our network is able to compute a semantic and instance
segmentation of the input image in a single forward pass.
We do not require any post-processing, such as the patch
aggregation of [ 37], “mask-voting” of [ 12], “superpixel

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


We do not require any post-processing, such as the patch
aggregation of [ 37], “mask-voting” of [ 12], “superpixel
projection” of [ 19,20,30] or spectral clustering of [ 33].
The fact that we compute an initial semantic segmentation
means that we have some robustness to errors in the object detector (Fig. 3). Furthermore, we are not necessarily
limited by poorly localised object detections either (Fig. 4).
Our CRF model allows us to reason about the entire image
at a time, rather than consider independent object proposals,
as done in [ 19,20,12,37,30]. Although we do not train our
object detector jointly with the network, it also means that
our segmentation network and object detector do not succumb to the same failure cases. Moreover, it ensures that
our instance labelling does not “switch” often during training, which makes learning more stable. Finally, note that
although we perform mean ﬁeld inference of a CRF within
our network, we do not optimise the CRF’s likelihood, but
rather a cross-entropy loss (Sec 3.4).
4. Experimental Evaluation
Sections 4.1to4.6describe our evaluation on the Pascal VOC Validation Set [ 14] and the Semantic Boundaries

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


4. Experimental Evaluation
Sections 4.1to4.6describe our evaluation on the Pascal VOC Validation Set [ 14] and the Semantic Boundaries
Dataset (SBD) [ 18] (which provides per-pixel annotations
to 11355 previously unlaballed images from Pascal VOC).
Section 4.7details results on Cityscapes [ 9].
4.1. Experimental Details
We ﬁrst train a network for semantic segmentation, therafter we ﬁnetune it to the task of instance segmentation,
as described in Sec. 3.5. Our training data for the semantic segmentation pretraining consists of images from Pascal VOC [ 14], SBD [ 18] and Microsoft COCO [ 35]. Finally, when ﬁnetuning for instance segmentation, we use
only training data from either the VOC dataset, or from the
SBD dataset. We train separate models for evaluating on
the VOC Validation Set, and the SBD Validation Set. In
each case, we remove validation set images from the initial
semantic segmentation pretraining set. We use the publicly
available R-FCN object detection framework [ 13], and ensure that the images used to train the detector do not fall
into our test sets for instance segmentation.
4.2. Evaluation Metrics
We report the mean Average Precision over regions

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


into our test sets for instance segmentation.
4.2. Evaluation Metrics
We report the mean Average Precision over regions
(APr) as deﬁned by [ 19]. The difference between APrand
the AP metric used in object detection [ 14] is that the Intersection over Union (IoU) is computed over predicted and
ground-truth regions instead of bounding boxes. Furthermore, the standard AP metric uses an IoU threshold of 0.5
to determine whether a prediction is correct or not. Here,
we use a variety of IoU thresholds since larger thresholds
require more precise segmentations. Additionally, we report theAPr
volwhich is the average of the APrfor 9 IoU
thresholds ranging from 0.1 to 0.9 in increments of 0.1.
However, we also observe that the APrmetric requires
an algorithm to produce a ranked list of segments and their
object class. It does not require, nor evaluate, the ability ofTable 1: The effect of the different CRF unary potentials,
and end-to-end training with them, on the VOC 2012 Validation Set.
APr
APr
volmatch
IoU 0.5 0.7 0.9
Box Term
(piecewise)60.0 47.3 21.2 54.9 42.6
Box+Global

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


IoU 0.5 0.7 0.9
Box Term
(piecewise)60.0 47.3 21.2 54.9 42.6
Box+Global
(piecewise)59.1 46.1 23.4 54.6 43.0
Box+Global+Shape
(piecewise)59.5 46.4 23.3 55.2 44.8
Box Term
(end-to-end)60.7 47.4 24.6 56.2 46.9
Box+Global
(end-to-end)60.9 48.1 25.5 56.7 47.1
Box+Global+Shape
(end-to-end)61.7 48.6 25.1 57.5 48.3
an algorithm to produce a globally coherent segmentation
map of the image, for example Fig. 1c. To measure this, we
propose the “Matching IoU” which matches the predicted
image and ground truth, and then calculates the corresponding IoU as deﬁned in [ 14]. This matching procedure is the
same as described in Sec. 3.4. This measure was originally
proposed in [ 54], but has not been used since in evaluating
instance segmentation systems.
4.3. Effect of Instance Potentials and End-to-End
training

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


proposed in [ 54], but has not been used since in evaluating
instance segmentation systems.
4.3. Effect of Instance Potentials and End-to-End
training
We ﬁrst perform ablation studies on the VOC 2012 Validation set. This dataset, consisting of 1464 training and
1449 validation images has very high-quality annotations
with detailed object delineations which makes it the most
suited for evaluating pixel-level segmentations.
In Tab. 1, we examine the effect of each of our unary potentials in our Instance subnetwork on overall performance.
Furthermore, we examine the effect of end-to-end training
the entire network as opposed to piecewise training. Piecewise training refers to freezing the pretrained semantic segmentation subnetwork’s weights and only optimising the instance segmentation subnetwork’s parameters. Note that
when training with only the “Box” (Eq. 3) unary potential and pairwise term, we also have to add in an additional
“Background” detection which encompasses the entire image. Otherwise, we cannot classify the background label.
We can see that each unary potential improves overall instance segmentation results, both in terms of APr
voland the
Matching IoU. The “Global” term (Eq. 4) shows particular

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


voland the
Matching IoU. The “Global” term (Eq. 4) shows particular
improvement over the “Box” term at the high APrthreshold of 0.9. This is because it can overcome errors in bounding box localisation (Fig. 4) and leverage our semantic segmentation network’s accurate predictions to produce precise
Table 2: Comparison of Instance Segmentation performance to recent methods on the VOC 2012 Validation Set
MethodAPr
APr
vol0.5 0.6 0.7 0.8 0.9
SDS [ 19] 43.8 34.5 21.3 8.7 0.9 –
Chen et al. [8] 46.3 38.2 27.0 13.5 2.6 –
PFN [ 33] 58.7 51.3 42.5 31.2 15.7 52.3
Arnab et al. [3] 58.3 52.4 45.4 34.9 20.1 53.1
MPA 1-scale [ 37] 60.3 54.6 45.9 34.3 17.3 54.5
MPA 3-scale [ 37]62.1 56.6 47.4 36.1 18.5 56.5

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


MPA 3-scale [ 37]62.1 56.6 47.4 36.1 18.5 56.5
Ours 61.7 55.5 48.6 39.5 25.1 57.5
labellings. The “Shape” term’s improvement in the APr
volis
primarily due to an improvement in the APrat low thresholds. By using shape priors, we are able to recover instances
which were occluded and missed out. End-to-end training
also improves results at all APrthresholds. Training with
just the “Box” term shows a modest improvement in the
APr
volof 1.3%. Training with the “Global” and “Shape”
terms shows larger improvements of 2.1% and 2.3% respectively. This may be because the “Box” term only considers
the semantic segmentation at parts of the image covered by
object detections. Once we include the “Global” term, we
consider the semantic segmentation over the entire image
for the detected class. Training makes more efﬁcient use of
images, and error gradients are more stable in this case.
4.4. Results on VOC Validation Set
We then compare our best instance segmentation model

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


images, and error gradients are more stable in this case.
4.4. Results on VOC Validation Set
We then compare our best instance segmentation model
to recent methods on the VOC Validation Set in Tab. 2. The
fact that our algorithm achieves the highest APrat thresholds above 0.7 indicates that our method produces more detailed and accurate segmentations.
At an IoU threshold of 0.9, our improvement over the
previous state-of-the-art (MPA [ 37]) is 6.6%, which is a
relative improvement of 36%. Unlike [ 37,19,8], our
network performs an initial semantic segmentation which
may explain our more accurate segmentations. Other
segmentation-based approaches, [ 3,33] are not fully endto-end trained. We also achieve the best APr
volof 57.5%.
The relatively small difference in APr
volto MPA [ 37] despite large improvements at high IoU thresholds indicates
that MPA performs better at low IoU thresholds. Proposalbased methods, such as [ 37,19] are more likely to perform
better at low IoU thresholds since they output more proposals than actual instances in an image (SDS evaluates 2000
proposals per image). Furthermore, note that whilst MPA

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


better at low IoU thresholds since they output more proposals than actual instances in an image (SDS evaluates 2000
proposals per image). Furthermore, note that whilst MPA
takes 8.7s to process an image [ 37], our method requires approximately 1.5s on the same Titan X GPU. More detailed
qualitative and quantitative results, including success and
failure cases, are included in the supplementary material.Table 3: Comparison of Instance Segmentation performance on the SBD Dataset
MethodAPr
APr
volmatch
0.5 0.7 IoU
SDS [ 19] 49.7 25.3 41.4 –
MPA 1-scale [ 37] 55.5 – 48.3 –
Hypercolumn [ 20] 56.5 37.0 – –
IIS [30] 60.1 38.7 – –
CFM [ 11] 60.7 39.6 – –
Hypercolumn rescore [ 20] 60.0 40.4 – –
MPA 3-scale rescore [ 37] 61.8 – 52.0 –
MNC [ 12] 63.5 41.5 – 39.0
MNC, Instance FCN [ 10] 61.5 43.0 – –
IIS sp. projection, rescore [ 30]63.6 43.3 – –

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


MNC, Instance FCN [ 10] 61.5 43.0 – –
IIS sp. projection, rescore [ 30]63.6 43.3 – –
Ours (piecewise) 59.1 42.1 52.3 41.8
Ours (end-to-end) 62.0 44.8 55.4 47.3
4.5. Results on SBD Dataset
We also evaluate our model on the SBD dataset, which
consists of 5623 training and 5732 validation images, as
shown in Tab. 3. Following other works, we only report
APrresults at IoU thresholds of 0.5 and 0.7. However, we
provide more detailed results in our supplementary material. Once again, we show signiﬁcant improvements over
other work at high APrthresholds. Here, our APrat 0.7
improves by 1.5% over the previous state-of-the-art [ 30].
Note that [ 30,37,20] perform additional post-processing
where their results are rescored using an additional object
detector. In contrast, our results are obtained by a single
forward pass through our network. We have also improved
substantially on the APr
volmeasure (3.4%) compared to

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


forward pass through our network. We have also improved
substantially on the APr
volmeasure (3.4%) compared to
other works which have reported it. We also used the publicly available source code1, model and default parameters
of MNC [ 12] to evaluate the “Matching IoU”. Our method
improves this by 8.3%. This metric is a stricter measure of
segmentation performance, and our method, which is based
on an initial semantic segmentation and includes a CRF as
part of training therefore performs better.
4.6. Improvement in Semantic Segmentation
Finetuning our network for instance segmentation, with
the loss described in Sec. 3.4improves semantic segmentation performance on both the VOC and SBD dataset, as
shown in Tab. 4. The improvement is 0.9% on VOC, and
1% on SBD. The tasks of instance segmentation and semantic segmentation are highly related – in fact, instance
segmentation can be thought of as a more speciﬁc case of
semantic segmentation. As a result, ﬁnetuning for one task
improves the other.
1https://github.com/daijifeng001/MNC
Table 4: Semantic Segmentation performance before and

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


improves the other.
1https://github.com/daijifeng001/MNC
Table 4: Semantic Segmentation performance before and
after ﬁnetuning for Instance Segmentation on the VOC and
SBD Validation Sets
DatasetMean IoU [%] before
Instance ﬁnetuningMean IoU [%] after
Instance ﬁnetuning
VOC 74.2 75.1
SBD 71.5 72.5
4.7. Results on Cityscapes
Finally, we evaluate our algorithm on the Cityscapes
road-scene understanding dataset [ 9]. This dataset consists
of 2975 training images, and the held-out test set consisting
of 1525 images are evaluated on an online server. None of
the 500 validation images were used for training. We use
an initial semantic segmentation subnetwork that is based
on the ResNet-101 architecture [ 59], and all of the instance
unary potentials described in Sec. 3.2.
As shown in Tab. 5, our method sets a new state-of-theart on Cityscapes, surpassing concurrent work [ 21] and the
best previous published work [ 49] by signiﬁcant margins.
Table 5: Results on Cityscapes Test Set. Evaluation metrics and results of competing methods obtained from the

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


best previous published work [ 49] by signiﬁcant margins.
Table 5: Results on Cityscapes Test Set. Evaluation metrics and results of competing methods obtained from the
online server. The “AP” metric of Cityscapes is similar to
ourAPr
volmetric.
Method AP AP at 0.5 AP 100m AP 50m
Ours 20.0 38.8 32.6 37.6
SAIS [ 21] 17.4 36.7 29.3 34.0
DWT [ 4] 15.6 30.0 26.2 31.8
InstanceCut [ 24] 13.0 27.9 22.1 26.1
Graph Decomp. [ 29] 9.8 23.2 16.8 20.3
RecAttend [ 43] 9.5 18.9 16.8 20.9
Pixel Encoding [ 49] 8.9 21.1 15.3 16.7
R-CNN [ 9] 4.6 12.9 7.7 10.3
5. Conclusion and Future Work
We have presented an end-to-end instance segmentation
approach that produces intermediate semantic segmentations, and shown that ﬁnetuning for instance segmentation

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


We have presented an end-to-end instance segmentation
approach that produces intermediate semantic segmentations, and shown that ﬁnetuning for instance segmentation
improves our network’s semantic segmentations. Our approach differs from other methods which derive their architectures from object detection networks [ 12,37,20] in that
our approach is more similar to a semantic segmentation
network. As a result, our system produces more accurate
and detailed segmentations as shown by our substantial improvements at high APrthresholds. Moreover, our system
produces segmentation maps naturally, and in contrast to
other published work, does not require any post-processing.
Finally, our network produces a variable number of outputs,depending on the number of instances in the image. Our future work is to incorporate an object detector into the endto-end training of our system to create a network that performs semantic segmentation, object detection and instance
segmentation jointly. Possible techniques for doing this are
suggested by [ 25] and [ 39].
Acknowledgements We thank Bernardino RomeraParedes and Stuart Golodetz for insightful discussions
and feedback. This work was supported by the EPSRC,
Clarendon Fund, ERC grant ERC-2012-AdG 321162HELIOS, EPRSRC grant Seebibyte EP/M013774/1 and

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


EPSRC/MURI grant EP/N019474/1.
References
[1] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik. Multiscale combinatorial grouping. In CVPR , pages
328–335. IEEE, 2014. 2
[2] A. Arnab, S. Jayasumana, S. Zheng, and P. H. S. Torr. Higher
order conditional random ﬁelds in deep neural networks. In
ECCV , 2016. 2,3,6
[3] A. Arnab and P. H. S. Torr. Bottom-up instance segmentation
with deep higher order crfs. In BMVC , 2016. 3,5,8,20
[4] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In arXiv preprint arXiv:1611.08303 ,
2016. 9
[5] L. Chen, A. Schwing, A. Yuille, and R. Urtasun. Learning
deep structured models. In ICML , Lille, France, 2015. 2
[6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


[6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR , 2015. 2
[7] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. L. Yuille. Deeplab: Semantic image segmentation with
deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915 , 2016. 1
[8] Y.-T. Chen, X. Liu, and M.-H. Yang. Multi-instance object segmentation with occlusion handling. In CVPR , pages
3470–3478, 2015. 2,4,5,8,20
[9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,
R. Benenson, U. Franke, S. Roth, and B. Schiele. The
cityscapes dataset for semantic urban scene understanding.
InCVPR , 2016. 1,7,9

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


cityscapes dataset for semantic urban scene understanding.
InCVPR , 2016. 1,7,9
[10] J. Dai, K. He, Y. Li, S. Ren, and J. Sun. Instance-sensitive
fully convolutional networks. In ECCV , 2016. 1,8
[11] J. Dai, K. He, and J. Sun. Convolutional feature masking for
joint object and stuff segmentation. In CVPR , 2015. 2,8
[12] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR , 2016. 1,2,
3,4,6,7,8,9,11,14,15
[13] J. Dai, Y. Li, K. He, and J. Sun. R-fcn: Object detection via
region-based fully convolutional networks. In NIPS , 2016.
1,7
[14] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and
A. Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010. 1,6,7

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


A. Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010. 1,6,7
[15] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained partbased models. PAMI , 32(9):1627–1645, 2010. 2,5
[16] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic
segmentation. In CVPR , 2014. 2
[17] S. Gupta, R. Girshick, P. Arbelaez, and J. Malik. Learning
rich features from RGB-D images for object detection and
segmentation. In ECCV . 2014. 1
[18] B. Hariharan, P. Arbel ´aez, L. Bourdev, S. Maji, and J. Malik.
Semantic contours from inverse detectors. In ICCV , pages
991–998. IEEE, 2011. 7
[19] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV , pages 297–

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


312. Springer, 2014. 1,2,3,4,6,7,8,20
[20] B. Hariharan, P. Arbel ´aez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and ﬁne-grained localization. CVPR , 2015. 1,2,3,4,6,7,8,9
[21] Z. Hayder, X. He, and M. Salzmann. Shape-aware instance
segmentation. In arXiv preprint arXiv:1612.03129 , 2016. 9
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR , 2016. 1
[23] X. He and S. Gould. An Exemplar-based CRF for Multiinstance Object Segmentation. In CVPR , 2014. 5
[24] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and
C. Rother. Instancecut: from edges to instances with multicut. In arXiv preprint arXiv:1611.08272 , 2016. 9
[25] I. Kokkinos. Ubernet: Training a universal convolutional

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


[25] I. Kokkinos. Ubernet: Training a universal convolutional
neural network for low-, mid-, and high-level vision using
diverse datasets and limited memory. In CVPR , 2017. 9
[26] P. Kr ¨ahenb ¨uhl and V. Koltun. Efﬁcient inference in fully
connected CRFs with Gaussian edge potentials. In NIPS ,
2011. 3,5
[27] P. Kr ¨ahenb ¨uhl and V. Koltun. Parameter learning and convergent inference for dense random ﬁelds. In ICML , 2013.
6
[28] L. Ladick ´y, P. Sturgess, K. Alahari, C. Russell, and P. H. S.
Torr. What, where and how many? combining object detectors and crfs. In ECCV , pages 424–437, 2010. 3
[29] E. Levinkov, S. Tang, E. Insafutdinov, and B. Andres. Joint
graph decomposition and node labeling by local search.
arXiv preprint arXiv:1611.04399 , 2016. 9

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


graph decomposition and node labeling by local search.
arXiv preprint arXiv:1611.04399 , 2016. 9
[30] K. Li, B. Hariharan, and J. Malik. Iterative Instance Segmentation. In CVPR , 2016. 1,2,4,6,7,8,21
[31] Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei. Fully convolutional
instance-aware semantic segmentation. In CVPR , 2017. 11,
16
[32] X. Liang, Y. Wei, X. Shen, Z. Jie, J. Feng, L. Lin, and S. Yan.
Reversible recursive instance-level object segmentation. In
CVPR , 2016. 1,2
[33] X. Liang, Y. Wei, X. Shen, J. Yang, L. Lin, and S. Yan.
Proposal-free network for instance-level object segmentation. arXiv preprint arXiv:1509.02636 , 2015. 2,6,8,18,
20
[34] G. Lin, C. Shen, and I. Reid. Efﬁcient piecewise training of
deep structured models for semantic segmentation. In CVPR ,

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


[34] G. Lin, C. Shen, and I. Reid. Efﬁcient piecewise training of
deep structured models for semantic segmentation. In CVPR ,
2016. 2[35] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740–755. Springer,
2014. 7,16
[36] M.-Y. Liu, O. Tuzel, A. Veeraraghavan, and R. Chellappa.
Fast directional chamfer matching. In CVPR , pages 1696–
1703. IEEE, 2010. 5
[37] S. Liu, X. Qi, J. Shi, H. Zhang, and J. Jia. Multi-scale patch
aggregation (mpa) for simultaneous detection and segmentation. In CVPR , 2016. 1,2,3,4,6,7,8,9,18,20
[38] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In CVPR , 2015. 3,6

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


[38] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. In CVPR , 2015. 3,6
[39] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Crossstitch Networks for Multi-task Learning. In CVPR , 2016. 9
[40] R. Pascanu, T. Mikolov, and Y. Bengio. On the difﬁculty of
training recurrent neural networks. In ICML , pages 1310–
1318, 2013. 6
[41] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In NIPS , 2015. 1
[42] P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollar. Learning to reﬁne object segments. In ECCV , 2016. 1
[43] M. Ren and R. S. Zemel. End-to-end instance segmentation and counting with recurrent attention. In arXiv preprint
arXiv:1605.09410 , 2016. 9

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


arXiv:1605.09410 , 2016. 9
[44] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS , 2015. 2
[45] B. Romera-Paredes and P. H. Torr. Recurrent instance segmentation. In ECCV , 2016. 2,6
[46] J. Shotton, A. Blake, and R. Cipolla. Contour-based learning
for object detection. In ICCV , volume 1, pages 503–510.
IEEE, 2005. 5
[47] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In ICLR , 2015.
3
[48] M. Sun, B.-s. Kim, P. Kohli, and S. Savarese. Relating things
and stuff via object property interactions. PAMI , 36(7):1370–
1383, 2014. 3
[49] J. Uhrig, M. Cordts, U. Franke, and T. Brox. Pixel-level encoding and depth layering for instance-level semantic labeling. In German Conference on Pattern Recognition (GCPR) ,
2016. 9

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


2016. 9
[50] D. Weiss and B. Taskar. Scalpel: Segmentation cascades
with localized priors and efﬁcient learning. In CVPR , pages
2035–2042, 2013. 5
[51] J. Winn and J. Shotton. The layout consistent random ﬁeld
for recognizing and segmenting partially occluded objects.
InCVPR , 2006. 2
[52] C. Wojek and B. Schiele. A dynamic conditional random
ﬁeld model for joint labeling of object and scene classes. In
ECCV , pages 733–747. Springer, 2008. 3
[53] N. Xu, B. Price, S. Cohen, J. Yang, and T. Huang. Deep
interactive object selection. In CVPR , 2016. 1
[54] Y. Yang, S. Hallman, D. Ramanan, and C. C. Fowlkes. Layered object models for image segmentation. PAMI , 2012. 2,
7
[55] J. Yao, S. Fidler, and R. Urtasun. Describing the Scene as
a Whole: Joint Object Detection, Scene Classiﬁcation and
Semantic Segmentation. In CVPR , pages 702–709, 2012. 3

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


a Whole: Joint Object Detection, Scene Classiﬁcation and
Semantic Segmentation. In CVPR , pages 702–709, 2012. 3
[56] S. Zagoruyko, A. Lerer, T.-Y. Lin, P. O. Pinheiro, S. Gross,
S. Chintala, and P. Doll ´ar. A multipath network for object
detection. In BMVC , 2016. 1
[57] Z. Zhang, S. Fidler, and R. Urtasun. Instance-level segmentation for autonomous driving with deep densely connected
mrfs. In CVPR , 2016. 2,3,4,6
[58] Z. Zhang, A. G. Schwing, S. Fidler, and R. Urtasun. Monocular object instance segmentation and depth ordering with
cnns. In ICCV , pages 2614–2622, 2015. 2,3,4,6
[59] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene
parsing network. In arXiv preprint arXiv:1612.01105 , 2017.
9
[60] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


9
[60] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,
Z. Su, D. Du, C. Huang, and P. Torr. Conditional random
ﬁelds as recurrent neural networks. In ICCV , 2015. 2,3,5,6
Appendix
In this supplementary material, we include more detailed
qualitative and quantitative results on the VOC and SBD
datasets. Furthermore, we also show the runtime of our algorithm.
Figures 7and8show success and failure cases of our
algorithm. Figure 9compares the results of our algorithm
to the publicly available model for MNC [ 12]. Figure 10
compares our results to those of FCIS [ 31], concurrent work
which won the COCO 2016 challenge. Figure 11presents
some qualitative results on the Cityscapes dataset.
Section Ashows more detailed results on the VOC
dataset. Figure 12shows a visualisation of our results at
differentAPrthresholds, and Tables 7to9show per-class
APrresults at thresholds of 0.5, 0.7 and 0.9.
Section Bshows more detailed results on the SBD

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


APrresults at thresholds of 0.5, 0.7 and 0.9.
Section Bshows more detailed results on the SBD
dataset. Table 6shows our mean APrresults at thresholds from 0.5 to 0.9, whilst Tables 10and11show per-class
APrresults at thresholds of 0.7 and 0.5 respectively.
Input image Semantic Segmentation Instance Segmentation Ground truth
Figure 7: Success cases of our method. First and second row: Our algorithm can leverage good initial semantic segmentations, and detections, to produce an instance segmentation. Third row: Notice that we have ignored three false-positive
detections. Additionally, the red bounding box does not completely encompass the person, but our algorithm is still able to
associate pixels “outside-the-box” with the correct detection (also applies to row 2). Fourth row: Our system is able to deal
with the heavily occluded sheep, and ignore the false-positive detection. Fifth row: We have not been able to identify one
bicycle on the left since it was not detected, but otherwise have performed well. Sixth row: Although subjective, the train has
not been annotated in the dataset, but both our initial semantic segmentation and object detection networks have identiﬁed it.

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


not been annotated in the dataset, but both our initial semantic segmentation and object detection networks have identiﬁed it.
Note that the ﬁrst three images are from the VOC dataset, and the last three from SBD. Annotations in the VOC dataset are
more detailed, and also make more use of the grey “ignore” label to indicate uncertain areas in the image. The ﬁrst column
shows the input image, and the results of our object detector which are another input to our network. Best viewed in colour.
Input image Semantic Segmentation Instance Segmentation Ground truth
Figure 8: Failure cases of our method. First row: Both our initial detector, and semantic segmentation system did not
identify a car in the background. Additionally, the “brown” person prediction actually consists of two people that have been
merged together. This is because the detector did not ﬁnd the background person. Second row: Our initial semantic segmentation identiﬁed the table, but it is not there in the Instance Segmentation. This is because there was no “table detection” to
associate these pixels with. Using heuristics, we could propose additional detections in cases like these. However, we have

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


associate these pixels with. Using heuristics, we could propose additional detections in cases like these. However, we have
not done this in our work. Third row: A difﬁcult case where we have segmented most of the people. However, sometimes
two people instances are joined together as one person instance. This problem is because we do not have a detection for
each person in the image. Fourth row: Due to our initial semantic segmentation, we have not been able to segment the green
person and table correctly. Fifth row: We have failed to segment a bird although it was detected. Sixth row: The occluding
cows, which all appear similar, pose a challenge, even with our shape priors. The ﬁrst column shows the input image, and
the results of our object detector which are another input to our network. Best viewed in colour.
Input image MNC [ 12] Ours Ground truth
Figure 9: Comparison to MNC [ 12]The above examples emphasise the advantages in our method over MNC [ 12]. Unlike
proposal-based approaches such as MNC, our method can handle false-positive detections, poor bounding box localisation,
reasons globally about the image and also produces more precise segmentations due to the initial semantic segmentation

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


reasons globally about the image and also produces more precise segmentations due to the initial semantic segmentation
module which includes a differentiable CRF. Row 1 shows a case where MNC, which scores segment-based proposals, is
fooled by a false-positive detection and segments an imaginary human (yellow segment). Our method is robust to falsepositive detections due to the initial semantic segmentation module which does not have the same failure modes as the
detector. Rows 2, 3 and 4 show how MNC [ 12] cannot deal with poorly localised bounding boxes. The horizontal boundaries
of the red person in Row 2, and light-blue person in Row 4 correspond to the limits of the proposal processed by MNC. Our
method, in contrast, can segment “outside the detection bounding box” due to the global instance unary potential (Eq. 4). As
MNC does not reason globally about the image, it cannot handle cases of overlapping bounding boxes well, and produces
more instances than there actually are. The ﬁrst column shows the input image, and the results of our object detector which
are another input to our network. MNC does not use these detections, but does internally produce box-based proposals which
are not shown. Best viewed in colour.
Input image MNC [ 12] Ours Ground truth

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


are not shown. Best viewed in colour.
Input image MNC [ 12] Ours Ground truth
Figure 9 continued: Comparison to MNC [ 12]The above examples show that our method produces more precise segmentations than MNC, that adhere to the boundaries of the objects. However, in Rows 3, 4 and 5, we see that MNC is able
to segment instances that our method misses out. In Row 3 , our algorithm does not segment the baby, although there is a
detection for it. This suggests that our shape prior which was formulated to overcome such occlusions could be better. As
MNC processes individual instances, it does not have a problem with dealing with small, occluding instances. In Row 4 ,
MNC has again identiﬁed a person that our algorithm could not. However, this is because we did not have a detection for
this person. In Row 5 , MNC has segmented the horses on the right better than our method. The ﬁrst column shows the input
image, and the results of our object detector which are another input to our network. MNC does not use these detections, but
does internally produce box-based proposals which are not shown. We used the publicly available code, models and default
parameters of MNC to produce this ﬁgure. Best viewed in colour.
Input image FCIS [ 31] Ours

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


parameters of MNC to produce this ﬁgure. Best viewed in colour.
Input image FCIS [ 31] Ours
Figure 10: Comparison to FCIS [ 31]The above images compare our method to the concurrent work, FCIS [ 31], which was
trained on COCO [ 35] and won the COCO 2016 challenge. Unlike proposal-based methods such as FCIS, our method can
handle false-positive detections and poor bounding-box localisation. Furthermore, as our method reasons globally about the
image, one pixel can only be assigned to a single instance, which is not the case with FCIS. Our method also produces more
precise segmentations, as it includes a differentiable CRF, and it is based off a semantic segmentation network. The results of
FCIS are obtained from their publicly available results on the COCO test set ( https://github.com/daijifeng001/
TA-FCN ). Note that FCIS is trained on COCO, and our model is trained on Pascal VOC which does not have as many classes
as COCO, such as “umbrella” and “suitcase” among others. As a result, we are not able to detect these objects. The ﬁrst

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


column shows the input image, and the results of our object detector which are another input to our network. FCIS does not
use these detections, but does internally produce proposals which are not shown. Best viewed in colour.
Input image Semantic Segmentation Instance Segmentation
Figure 11: Sample results on the Cityscapes dataset The above images show how our method can handle the large numbers
of instances present in the Cityscapes dataset. Unlike other recent approaches, our algorithm can deal with objects that are
not continuous – such as the car in the ﬁrst row which is occluded by a pole. Best viewed in colour.
A. Detailed results on the VOC dataset
Figure 12shows a visualisation of the AProbtained by
our method for each class across nine different thresholds.
Each “column” of Fig. 12corresponds to the APrfor each
class at a given IoU threshold. It is therefore an alternate
representation for the results tables (Tables 7to9). We can
see that our method struggles with classes such as “bicycle”, “chair”, “dining table” and “potted plant”. This may
be explained by the fact that current semantic segmentation

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


be explained by the fact that current semantic segmentation
systems (including ours) struggle with these classes. All recent methods on the Pascal VOC leaderboard2obtain an
IoU for these classes which is lower than the mean IoU for
all classes. In fact the semantic segmentation IoU for the
“chair” class is less than half of the mean IoU for all the
classes for 16 out of the 20 most recent submissions on the
VOC leaderboard at the time of writing.
Tables 7to9show per-class instance segmentation results on the VOC dataset, at IoU thresholds of 0.9, 0.7 and
0.5 respectively. At an IoU threshold of 0.9, our method
achieves the highest APrfor 16 of the 20 object classes.
At the threshold of 0.7, we achieve the highest APrin 15
classes. Finally, at an IoU threshold of 0.5, our method,
MPA 3-scale [ 37] and PFN [ 33] each achieve the highest
APrfor 6 categories.
B. Detailed results on the SBD dataset
Once again, we show a visualisation of the AProbtained
by our method for each class across nine different thresholds (Fig. 13). The trend is quite similar to the VOC dataset
in that our algorithm struggles on the same object classes

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


by our method for each class across nine different thresholds (Fig. 13). The trend is quite similar to the VOC dataset
in that our algorithm struggles on the same object classes
(“chair”, “dining table”, “potted plant”, “bottle”). Note that
ourAPrfor the “bicycle” class has improved compared to
the VOC dataset. This is probably because the VOC dataset
has more detailed annotations. In the VOC dataset, each
spoke of a bicycle’s wheel is often labelled, whilst in SBD,
the entire wheel is labelled as a single circle with the “bicycle” label. Therefore, the SBD dataset’s coarser labelling
makes it easier for an algorithm to perform well on objects
with ﬁne details.
Table 6shows our mean AProver all classes at thresholds ranging from 0.5 to 0.9. Our APrat 0.9 is low compared to the result which we obtained on the VOC dataset.
This could be for a number of reasons: As the SBD dataset
is not as ﬁnely annotated as the VOC dataset, it might not
be suited for measuring the APrat such high thresholds.
Additionally, the training data is not as good for training

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


be suited for measuring the APrat such high thresholds.
Additionally, the training data is not as good for training
our system which includes a CRF and is therefore able to
delineate sharp boundaries. Finally, as the SBD dataset has
5732 validation images (compared to the 1449 in VOC), it
leaves less data for pretraining our initial semantic segmen2http://host.robots.ox.ac.uk:8080/leaderboard/
displaylb.php?challengeid=11&compid=6tation module. This may hinder our network in being able
to produce precise segmentations.
Table 6: Comparison of Instance Segmentation performance at multiple APrthesholds on the VOC 2012 Validation Set
MethodAPr
APr
vol0.5 0.6 0.7 0.8 0.9
Ours (piecewise) 59.1 51.9 42.1 29.4 12.0 52.3
Ours (end-to-end ) 62.0 54.0 44.8 32.3 13.8 55.4
Tables 10and11show per-class instance segmentation
results on the SBD dataset, at IoU thresholds of 0.7 and
0.5 respectively. We can only compare results at these two
thresholds since these are the only thresholds which other

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


0.5 respectively. We can only compare results at these two
thresholds since these are the only thresholds which other
work has reported.
Figure 12: A visualisation of the AProbtained for each of the 20 classes on the VOC dataset, at nine different IoU thresholds.
The x-axis represents the IoU threshold, and the y-axis each of the Pascal classes. Therefore, each “column” of this ﬁgure
corresponds to the APrper class at a particular threshold, and is thus an alternate representation to the results tables. Best
viewed in colour.
Figure 13: A visualisation of the AProbtained for each of the 20 classes on the SBD dataset, at nine different IoU thresholds.
The x-axis represents the IoU threshold, and the y-axis each of the Pascal classes. Therefore, each “column” of this ﬁgure
corresponds to the APrper class at a particular threshold, and is thus an alternate representation to the results tables. Best
viewed in colour.
Table 7: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.9, for all twenty classes in the VOC dataset.
MethodMean

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


Table 7: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.9, for all twenty classes in the VOC dataset.
MethodMean
APr(%)aeroplanebike bird boatbottlebus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv
Our method 25.1 56.6 0.03 36.8 14.4 9.9 39.0 17.2 47.1 1.3 29.0 9.5 47.2 29.8 20.0 14.8 2.3 25.9 23.8 45.7 32.3
MPA 3-scale [ 37] 18.5 – – – – – – – – – – – – – – – – – – – –
MPA 1-scale [ 37] 17.3 – – – – – – – – – – – – – – – – – – – –
Arnab et al. [3] 20.1 43.7 0.03 30.0 13.2 11.4 47.3 10.9 34.5 0.7 19.6 12.1 35.6 24.3 13.3 10.7 0.4 20.7 20.9 35.0 17.4

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


PFN [ 33] 15.7 43.9 0.1 24.5 7.8 4.1 32.5 6.3 42.0 0.6 25.7 3.2 31.8 13.4 8.1 5.9 1.6 14.8 14.3 25.0 8.5
Chen et al. [8] 2.6 0.6 0 0.6 0.5 4.9 9.8 1.1 8.3 0.1 1.1 1.2 1.7 0.3 0.8 0.6 0.3 0.8 7.6 4.3 6.2
SDS [ 19] 0.9 0 0 0.2 0.3 2.0 3.8 0.2 0.9 0.1 0.2 1.5 0 0 0 0.1 0.1 0 2.3 0.2 5.8
Table 8: Comparison of mean APr, achieved by different published methods, at an IoU threshold of 0.7, for all twenty classes in the VOC dataset.
MethodMean
APr(%)aeroplanebike bird boatbottlebus car cat chair cow table dog horse mbikepersonplant sheep sofa train tv

CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models


CVPR 2017

1. Multiscale Combinatorial Grouping

5. Learning Deep Structured Models