Different channels appear to capture the motion of different body parts. In the 5thcolumn, we display
the offset magnitudes, which highlight salient human motion. Finally, the last two columns illustrate
the standard Farneback ﬂow, and the human motion predicted from our learned offsets. To predict
human motion we train a linear classiﬁer to regress the ground-truth (x;y)displacement of each
joint from the offset maps. The color wheel, at the bottom right corner encodes motion direction.
4.5 Interpreting Learned Offsets
Understanding what information is encoded in our learned offsets is nearly as difﬁcult as analyzing
any other CNN features [ 59,60]. The main challenge comes from the high dimensionality of offsets:
we are predicting ckhkw(x;y)displacements for every pixel for each of the ﬁve dilation
ratesd, wherecis the number of channels, and kh;kware the convolutional kernel height and width
respectively.
In columns 3;4of Figure 5, we visualize two randomly-selected offset channels as a motion ﬁeld.
Based on this ﬁgure, it appears that different offset maps encode different motions rather than all

Accepted to NeurIPS 2019

6. Mask R-CNN


Based on this ﬁgure, it appears that different offset maps encode different motions rather than all
predicting the same solution (say, the optical ﬂow between the two frames). This makes sense, as the
network may decide to ignore motions of uninformative regions, and instead capture the motion of
different human body parts in different offset maps (say, a hand as opposed to the head). We also note
that the magnitudes of our learned offsets encode salient human motion (see Column 5 of Figure 5).
Lastly, to verify that our learned offsets encode human motion, for each point pndenoting a body joint,
we extract our predicted offsets and train a linear classiﬁer to regress the ground truth (x;y)motion
displacement of this body joint. In Column 7 of Figure 5, we visualize our predicted motion outputs
for every pixel. We show Farneback’s optical ﬂow in Column 6. Note that in regions containing
people, our predicted human motion matches Farneback optical ﬂow. Furthermore, we point out that
compared to the standard Farneback optical ﬂow, our motion ﬁelds look less noisy.
5 Conclusions
In this work, we introduced PoseWarper, a novel architecture for pose detection in sparsely labeled

Learning Temporal Pose Estimation from Sparsely-Labeled Videos

Summary:
Modern approaches for multi-person pose estimation in video require large
amounts of dense annotations. However, labeling every frame in a video is
costly and labor intensive. To reduce the need for dense annotations, we
propose a PoseWarper network that leverages training videos with sparse
annotations (every k frames) to learn to perform dense temporal pose
propagation and estimation. Given a pair of video frames---a labeled Frame A
and an unlabeled Frame B---we train our model to predict human pose in Frame A
using the features from Frame B by means of deformable convolutions to
implicitly learn the pose warping between A and B. We demonstrate that we can
leverage our trained PoseWarper for several applications. First, at inference
time we can reverse the application direction of our network in order to
propagate pose information from manually annotated frames to unlabeled frames.
This makes it possible to generate pose annotations for the entire video given
only a few manually-labeled frames. Compared to modern label propagation
methods based on optical flow, our warping mechanism is much more compact (6M
vs 39M parameters), and also more accurate (88.7% mAP vs 83.8% mAP). We also
show that we can improve the accuracy of a pose estimator by training it on an
augmented dataset obtained by adding our propagated poses to the original
manual labels. Lastly, we can use our PoseWarper to aggregate temporal pose
information from neighboring frames during inference. This allows our system to
achieve state-of-the-art pose detection results on the PoseTrack2017 and
PoseTrack2018 datasets. Code has been made available at:
https://github.com/facebookresearch/PoseWarper.

Gedas Bertasius
Christoph Feichtenhofer
Du Tran
Jianbo Shi
Lorenzo Torresani

Accepted to NeurIPS 2019

1. DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection

Gedas Bertasius, Jianbo Shi, Lorenzo Torresani

2. Aggregated Residual Transformations for Deep Neural Networks

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, Kaiming He

3. DeepPose: Human Pose Estimation via Deep Neural Networks

4. Convolutional Random Walk Networks for Semantic Image Segmentation

Gedas Bertasius, Lorenzo Torresani, Stella X. Yu, Jianbo Shi

5. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

6. Mask R-CNN

Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick


5 Conclusions
In this work, we introduced PoseWarper, a novel architecture for pose detection in sparsely labeled
videos. Our PoseWarper can be effectively used for multiple applications, including video pose
propagation, and temporal pose aggregation. In these settings, we demonstrated that our approach
reduces the need for densely labeled video data, while producing strong pose detection performance.
Furthermore, our state-of-the-art results on PoseTrack2017 and PoseTrack2018 datasets demonstrate
that our PoseWarper is useful even when the training videos are densely-labeled. Our future work
involves improving our model ability to propagate labels and aggregate temporal information when
the input frames are far away from each other. We are also interested in exploring self-supervised
learning objectives for our task, which may further reduce the need of pose annotations in video. We
will release our source code and our trained models upon publication of the paper.
References
[1]Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Deepedge: A multi-scale bifurcated deep network
for top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) , June 2015.
9

Accepted to NeurIPS 2019

6. Mask R-CNN


for top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) , June 2015.
9
[2]Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR , 2017.
[3]Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In
CVPR , 2014.
[4]Gedas Bertasius, Lorenzo Torresani, Stella X. Yu, and Jianbo Shi. Convolutional random walk networks
for semantic image segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) , July 2017.
[5]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional
networks for visual recognition. In Computer Vision – ECCV 2014 , 2014.
[6]Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the
International Conference on Computer Vision (ICCV) , 2017.

Accepted to NeurIPS 2019

6. Mask R-CNN


International Conference on Computer Vision (ICCV) , 2017.
[7]Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object
Detection. In Proceedings of the International Conference on Computer Vision (ICCV) , 2017.
[8]Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object
detection with region proposal networks. In Neural Information Processing Systems (NIPS) , 2015.
[9]Ross Girshick. Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV) ,
2015.
[10] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate
object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) , 2014.
[11] Saurabh Gupta, Ross Girshick, Pablo Arbelaez, and Jitendra Malik. Learning rich features from RGB-D
images for object detection and segmentation. In ECCV , 2014.
[12] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and

Accepted to NeurIPS 2019

6. Mask R-CNN


[12] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and
Alexander C. Berg. Ssd: Single shot multibox detector. In ECCV , 2016.
[13] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fcn: Object detection via region-based fully convolutional
networks. In Advances in Neural Information Processing Systems 29 , pages 379–387. Curran Associates,
Inc., 2016.
[14] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Uniﬁed,
real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2016, Las Vegas, NV , USA, June 27-30, 2016 , pages 779–788, 2016.
[15] Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on
Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages
6517–6525, 2017.

Accepted to NeurIPS 2019

6. Mask R-CNN


Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages
6517–6525, 2017.
[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classiﬁcation with deep convolutional
neural networks. In NIPS , 2012.
[17] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru
Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Computer Vision
and Pattern Recognition (CVPR) , 2015.
[18] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In
ICLR , 2015.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770–778, 2016.
[20] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image

Accepted to NeurIPS 2019

6. Mask R-CNN


Database. In CVPR09 , 2009.
[21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár,
and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on
Computer Vision (ECCV) , Zürich, September 2014.
[22] Umar Iqbal, Anton Milan, and Juergen Gall. Posetrack: Joint multi-person pose estimation and tracking.
InIEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.
10
[23] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In
European Conference on Computer Vision (ECCV) , 2018.
[24] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using
part afﬁnity ﬁelds. In CVPR , 2017.
[25] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In
Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14,

Accepted to NeurIPS 2019

6. Mask R-CNN


Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14,
2016, Proceedings, Part VIII , pages 483–499, 2016.
[26] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In
CVPR , pages 4724–4732. IEEE Computer Society, 2016.
[27] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human
pose estimation. In CVPR , 2019.
[28] J. Dai, H. Qi, Y . Xiong, Y . Li, G. Zhang, H. Hu, and Y . Wei. Deformable convolutional networks. In 2017
IEEE International Conference on Computer Vision (ICCV) , volume 00, pages 764–773, Oct. 2017.
[29] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.
Flownet 2.0: Evolution of optical ﬂow estimation with deep networks. CoRR , abs/1612.01925, 2016.
[30] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Pictorial structures revisited: People detection and

Accepted to NeurIPS 2019

6. Mask R-CNN


[30] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Pictorial structures revisited: People detection and
articulated pose estimation. In CVPR , pages 1014–1021. IEEE Computer Society, 2009.
[31] M. Andriluka, S. Roth, and B. Schiele. Monocular 3d pose estimation and tracking by detection. In 2010
IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 623–630, June
2010.
[32] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose
estimation. In Proc. BMVC , pages 12.1–11, 2010. doi:10.5244/C.24.12.
[33] Leonid Pishchulin, Mykhaylo Andriluka, Peter V . Gehler, and Bernt Schiele. Strong appearance and
expressive spatial models for human pose estimation. In ICCV , pages 3487–3494. IEEE Computer Society,
2013.
[34] Yi Yang and D. Ramanan. Articulated pose estimation with ﬂexible mixtures-of-parts. In Proceedings of
the 2011 IEEE Conference on Computer Vision and Pattern Recognition , CVPR ’11, pages 1385–1392,

Accepted to NeurIPS 2019

6. Mask R-CNN


the 2011 IEEE Conference on Computer Vision and Pattern Recognition , CVPR ’11, pages 1385–1392,
Washington, DC, USA, 2011. IEEE Computer Society.
[35] M. Dantone, J. Gall, C. Leistner, and L. van Gool. Human pose estimation using body parts dependent
joint regressors. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pages 3041–3048,
Portland, OR, USA, June 2013. IEEE.
[36] Xiangyang Lan and D. P. Huttenlocher. Beyond trees: common-factor models for 2d human pose recovery.
InTenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1 , volume 1, pages
470–477 V ol. 1, Oct 2005.
[37] L. Sigal and M. J. Black. Measure locally, reason globally: Occlusion-sensitive articulated pose estimation.
InProc. IEEE Conf. on Computer Vision and Pattern Recognition, CVPR , volume 2, pages 2041–2048,
New York, NY , June 2006.
[38] Yang Wang and Greg Mori. Multiple tree models for occlusion and spatial constraints in human pose
estimation. In David Forsyth, Philip Torr, and Andrew Zisserman, editors, Computer Vision – ECCV 2008 ,

Accepted to NeurIPS 2019

6. Mask R-CNN


estimation. In David Forsyth, Philip Torr, and Andrew Zisserman, editors, Computer Vision – ECCV 2008 ,
pages 710–724, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg.
[39] J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. Human pose estimation with iterative error feedback.
In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4733–4742, June
2016.
[40] Xianjie Chen and Alan L Yuille. Articulated pose estimation by a graphical model with image dependent
pairwise relations. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger,
editors, Advances in Neural Information Processing Systems 27 , pages 1736–1744. Curran Associates,
Inc., 2014.
[41] W. Ouyang, X. Chu, and X. Wang. Multi-source deep learning for human pose estimation. In 2014 IEEE
Conference on Computer Vision and Pattern Recognition , pages 2337–2344, June 2014.
11

Accepted to NeurIPS 2019

6. Mask R-CNN


Conference on Computer Vision and Pattern Recognition , pages 2337–2344, June 2014.
11
[42] Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schieke. Deepercut:
A deeper, stronger, and faster multi-person pose estimation model. In European Conference on Computer
Vision (ECCV) , 2016.
[43] Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, Peter Gehler, and
Bernt Schiele. Deepcut: Joint subset partition and labeling for multi person pose estimation. In IEEE
Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.
[44] Jonathan J Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of a convolutional
network and a graphical model for human pose estimation. In Z. Ghahramani, M. Welling, C. Cortes, N. D.
Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27 , pages
1799–1807. Curran Associates, Inc., 2014.

Accepted to NeurIPS 2019

6. Mask R-CNN


Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27 , pages
1799–1807. Curran Associates, Inc., 2014.
[45] George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and
Kevin Murphy. Towards accurate multi-person pose estimation in the wild. In CVPR , pages 3711–3719.
IEEE Computer Society, 2017.
[46] Vasileios Belagiannis and Andrew Zisserman. Recurrent human pose estimation. In International
Conference on Automatic Face and Gesture Recognition . IEEE, 2017.
[47] Eldar Insafutdinov, Mykhaylo Andriluka, Leonid Pishchulin, Siyu Tang, Evgeny Levinkov, Bjoern Andres,
and Bernt Schiele. Articulated multi-person tracking in the wild. In 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) , pages 1293–1301. IEEE, July 2017. Oral.
[48] Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri, and Du Tran. Detect-and-Track:
Efﬁcient Pose Estimation in Videos. In CVPR , 2018.

Accepted to NeurIPS 2019

6. Mask R-CNN


Efﬁcient Pose Estimation in Videos. In CVPR , 2018.
[49] Jie Song, Limin Wang, Luc Van Gool, and Otmar Hilliges. Thin-slicing network: A deep structured model
for pose estimation in videos. In CVPR , pages 4420–4229, 2017.
[50] J. Charles, T. Pﬁster, D. Magee, D. Hogg, and A. Zisserman. Personalizing human video pose estimation.
InIEEE Conference on Computer Vision and Pattern Recognition , 2016.
[51] T. Pﬁster, J. Charles, and A. Zisserman. Flowing convnets for human pose estimation in videos. In IEEE
International Conference on Computer Vision , 2015.
[52] Dingwen Zhang, Guangyu Guo, Dong Huang, and Junwei Han. Poseﬂow: A deep motion representation
for understanding human behaviors in videos. In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) , June 2018.
[53] O. Wiles, A.S. Koepke, and A. Zisserman. Self-supervised learning of a facial attribute embedding from
video. In British Machine Vision Conference , 2018.

Accepted to NeurIPS 2019

6. Mask R-CNN


video. In British Machine Vision Conference , 2018.
[54] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.
[55] Gunnar Farnebäck. Two-frame motion estimation based on polynomial expansion. In Proceedings of the
13th Scandinavian Conference on Image Analysis , SCIA’03, pages 363–370, Berlin, Heidelberg, 2003.
Springer-Verlag.
[56] Hengkai Guo, Tang Tang, Guozhong Luo, Riwei Chen, Yongchen Lu, and Linfu Wen. Multi-domain pose
network for multi-person pose estimation and tracking. In Computer Vision - ECCV 2018 Workshops Munich, Germany, September 8-14, 2018, Proceedings, Part II , pages 209–216, 2018.
[57] Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, and Cewu Lu. Pose Flow: Efﬁcient online pose
tracking. In BMVC , 2018.
[58] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estimation. In
The IEEE International Conference on Computer Vision (ICCV) , Oct 2017.

Accepted to NeurIPS 2019

6. Mask R-CNN


The IEEE International Conference on Computer Vision (ICCV) , Oct 2017.
[59] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. CoRR ,
abs/1311.2901, 2013.
[60] Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson. Understanding neural
networks through deep visualization. CoRR , abs/1506.06579, 2015.
12

Accepted to NeurIPS 2019

6. Mask R-CNN


Classifying, Segmenting, and Tracking
Object Instances in Video with Mask Propagation
Gedas Bertasius, Lorenzo Torresani
Facebook AI
Abstract
We introduce a method for simultaneously classifying,
segmenting and tracking object instances in a video sequence. Our method, named MaskProp, adapts the popular Mask R-CNN to video by adding a mask propagation
branch that propagates frame-level object instance masks
from each video frame to all the other frames in a video clip.
This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip. Clip-level instance tracks generated
densely for each frame in the sequence are ﬁnally aggregated to produce video-level object instance segmentation
and classiﬁcation. Our experiments demonstrate that our
clip-level instance segmentation makes our approach robust
to motion blur and object occlusions in video. MaskProp
achieves the best reported accuracy on the YouTube-VIS
dataset, outperforming the ICCV 2019 video instance segmentation challenge winner despite being much simpler and
using orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes).
1. Introduction

Summary:
We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).

CVPR 2020 Best Paper Nominee

1. Pixelwise Instance Segmentation with a Dynamically Instantiated Network

2. Deep Watershed Transform for Instance Segmentation

3. Learning Temporal Pose Estimation from Sparsely Labeled Videos

Gedas Bertasius, Christoph Feichtenhofer, Du Tran, Jianbo Shi, Lorenzo Torresani

4. Semantic Segmentation with Boundary Neural Fields

Gedas Bertasius, Lorenzo Torresani, Jianbo Shi

5. Object Detection in Video with Spatiotemporal Sampling Networks


using orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes).
1. Introduction
In this paper, we tackle the recently introduced video instance segmentation problem [42]. This task requires segmenting all instances of a predeﬁned set of object classes
in each frame, classifying them, and linking individual instances over the entire sequence.
In recent years, convolutional networks have obtained remarkable results in still-image object detection [16, 33, 14,
15], and segmentation [27, 45, 8, 4]. However, extending
these models to video instance segmentation is challenging. In order to localize objects precisely, these methods
have to operate at very large spatial resolution. As a result,
detectors based on the popular ResNet-101 or ResNet-152
backbones [17] can rarely ﬁt more than one image per GPU
during training. In the context of video instance segmentation this is problematic because tracking objects over time
requires analyzing multiple video frames simultaneously.
To address this issue, one could reduce the spatial resFigure 1: In this paper, we tackle the problem of video instance segmentation, which requires classifying, segmenting, and tracking object instances in a given video sequence.
Our proposed Mask Propagation framework (MaskProp)
provides a simple and effective way for solving this task.

CVPR 2020 Best Paper Nominee


Our proposed Mask Propagation framework (MaskProp)
provides a simple and effective way for solving this task.
olution of the input and ﬁt more video frames in a GPU.
However, doing so typically leads to a signiﬁcant drop in
segmentation or detection performance. Alternatively, one
could perform high-resolution instance segmentation on individual frames and then link segmentations temporally in
a separate post-processing stage. However, performing instance segmentation and tracking in two disjoint steps often produces suboptimal results, because these two tasks
are closely intertwined. The key challenge then becomes
designing a uniﬁed model that can track objects in video
while maintaining strong detection accuracy.
Currently, the best method for video instance segmentation is the ICCV 2019 challenge winner [28]. It tackles
video instance segmentation by dividing it into four problems: 1) detection, 2) classiﬁcation, 3) segmentation, and
4) tracking. These four problems are solved independently
using several off-the-shelf components and their respective
solutions are combined and adapted to the video instance
segmentation task. However, despite effective performance,
such an approach is disadvantageous because it requires designing and tuning a separate model (or, in some cases, an

CVPR 2020 Best Paper Nominee


segmentation task. However, despite effective performance,
such an approach is disadvantageous because it requires designing and tuning a separate model (or, in some cases, an
ensemble of models) for each of the four tasks. This renders the approach costly and cumbersome. On the other
end of the complexity spectrum, MaskTrack R-CNN [42] is
a simple uniﬁed approach trained end-to-end but it achieves
signiﬁcantly lower performance ( 30:3vs44:8video mAP).
To address the shortcomings of these prior methods wearXiv:1912.04573v4  [cs.CV]  9 Jul 2021
MaskTrack R-CNN [42] ICCV19 Challenge Winner [28] MaskProp
ModelClassiﬁcation cls head Mask R-CNN [16], ResNeXt-101 32x48d [30] cls head
Localization bbox head Mask R-CNN [16] bbox head
Segmentation mask head DeepLabv3 [9], Box2Seg [29] mask head
Tracking tracking head UnOVOST [47], ReID Net [18, 31] mask propagation head
Optical Flow - PWC-Net [35] Pre-training DatasetsImageNet [34] (1.3M images) X X X

CVPR 2020 Best Paper Nominee


Optical Flow - PWC-Net [35] Pre-training DatasetsImageNet [34] (1.3M images) X X X
COCO [25] (860K bboxes) X X X
Instagram [30] (1B images) - X OpenImages [23] (14M bboxes) - X Performancevideo mAP 30.3 44.8 46.6
video AP@75 32.6 48.9 51.2
Table 1: A table comparing our work to prior video instance segmentation methods [42, 28]. The ICCV 2019 Challenge
Winner [28] decomposes video instance segmentation into four different problems, solves each of them independently using
ensembles of different models, and then combines these solutions. In contrast, our approach relies on a single uniﬁed model
trained end-to-end. Despite being simpler, and using several orders of magnitude less pretraining data (1.3M vs 1B images
and 860K vs 14M bounding boxes) our model achieves higher accuracy. Furthermore, compared to MaskTrack R-CNN [42],
our work yields a 16.3% gain in mAP (46.6% vs 30.3%).
introduce MaskProp, a simple mask propagation framework
for simultaneously classifying, segmenting and tracking object instances in video. Our method adapts the popular

CVPR 2020 Best Paper Nominee


introduce MaskProp, a simple mask propagation framework
for simultaneously classifying, segmenting and tracking object instances in video. Our method adapts the popular
Mask R-CNN [16] to video by adding a branch that propagates frame-level instance masks from each video frame
to other frames within a temporal neighborhood (which we
refer to as a clip). This allows our method to compute cliplevel instance tracks centered at each individual frame of the
video. These densely estimated clip-level tracks are then
aggregated to form accurate and coherent object instance
sequences for the entire video, regardless of its length. This
renders our approach capable of handling challenging cases
of occlusions, disocclusions, and motion blur. Our method
achieves the best reported accuracy on the YouTube-VIS
dataset [42], outperforming the ICCV 2019 challenge winner [28] despite being much simpler and using signiﬁcantly
less labeled data (1000x fewer images and 10x fewer bounding boxes). In Table 1, we compare our approach vs these
prior methods in terms of accuracy and other characteristics.
2. Related Work
Instance Segmentation in Images. Compared to instance
segmentation in images [11, 16, 44, 1, 2, 22, 26], the problem considered in this paper requires not only to segment
object instances in individual frames, but also to determine

CVPR 2020 Best Paper Nominee


object instances in individual frames, but also to determine
instance correspondences across multiple frames. We leverage the Mask R-CNN model [16] for still-image instance
segmentation and adapt it to track object instances in video.
Object Detection in Video. Object detection in video requires classifying and localizing objects in every frame of
a given video. Most modern video object detection systems [46, 5, 40, 13] implement some form of spatiotemporal feature alignment for improving object detection accuracy in individual video frames. However, these systems
are typically not designed for tracking object instances. In
contrast, our mask propagation produces clip-level instancesegmentations rather than frame-level bounding boxes.
Video Object Segmentation. The task of video object
segmentation requires segmenting foreground objects in a
class-agnostic fashion [43, 21, 36, 38], often by leveraging
ground truth masks available for the ﬁrst frame during inference [6, 32, 10, 19, 37]. Instead, video instance segmentation requires ﬁnding all instances of a predeﬁned set of
object classes in each frame, classifying them and linking
them over the entire sequence.
Video Instance Segmentation. The recently introduced
video instance segmentation task [42] requires classifying,
segmenting and tracking object instances in videos. This

CVPR 2020 Best Paper Nominee


Video Instance Segmentation. The recently introduced
video instance segmentation task [42] requires classifying,
segmenting and tracking object instances in videos. This
is the task considered in this work. There are only a few
video instance segmentation methods we can compare our
approach to. The MaskTrack R-CNN [42] presents a uniﬁed model for video instance segmentation. It augments the
original Mask R-CNN [16] with a tracking branch that establishes associations among object instances segmented in
separate frames. Furthermore, we include the ICCV 2019
video instance segmentation challenge winner [28] in our
comparison. This approach divides video instance segmentation into four separate subproblems: classiﬁcation, detection, segmentation, and tracking. A separate model (or
an ensemble of models) is used to solve each of these subproblems, and these solutions are then combined to produce
video instance segmentation results. For brevity, from now
on we refer to it as EnsembleVIS to indicate that it is an ensemble approach designed for video instance segmentation.
Our MaskProp framework provides advantages over
both of these methods [42, 28]. Similarly to MaskTrack
R-CNN [42], our method is a uniﬁed and simple approach.

CVPR 2020 Best Paper Nominee


both of these methods [42, 28]. Similarly to MaskTrack
R-CNN [42], our method is a uniﬁed and simple approach.
However, our mask propagation branch is much more effective than the tracking branch of MaskTrack R-CNN, achieving much higher accuracy relative to this baseline. Furthermore, compared to EnsembleVIS [28], our method 1)
is much simpler, 2) uses signiﬁcantly less labeled data, and

CVPR 2020 Best Paper Nominee


clip-level     instance tracktemporally propagated instance featuresfeature tensorinstance feature computation at time t    frame-level    instance mask time ttime t⌦<latexit

CVPR 2020 Best Paper Nominee


CVPR 2020 Best Paper Nominee


time ttime t+10time t-5time t+5time t+10time ttime t-10propagated instance        segmentationfeature tensors for each frame             in a video clip
time t-5time t+5time t+10time ttime t-10
Figure 2: An illustration of our MaskProp system, which takes as input a video clip centered around frame t, and outputs
a clip-level instance track. Our mask propagation framework can be summarized in three high-level steps: 1) An instancespeciﬁc feature map at time tis computed by masking the frame features at time twith the given instance segmentation
for framet(one for each instance detected in frame t). 2) Next, we use our mask propagation mechanism to temporally
propagate instance-speciﬁc features from frame tto all the other frames in the clip. 3) Lastly, our model predicts instancespeciﬁc segmentations in every frame of the clip by implicitly matching the propagated instance features with the frame-level
features computed at each time step. This last step yields clip-level instance tracks centered around frame t.
3) produces higher accuracy on YouTube-VIS [42].
3. Video Instance Segmentation
Problem Deﬁnition. Let us denote with V2RL3HW

CVPR 2020 Best Paper Nominee


3. Video Instance Segmentation
Problem Deﬁnition. Let us denote with V2RL3HW
an input video consisting of LRGB frames of spatial size
HW. The aim of our system is to segment and temporally
link all object instances that are visible for at least one frame
inVand that belong to a predeﬁned set of categories C=
f1;:::;Kg. To achieve this goal, our model outputs a videolevel instance mask track Mi2RLHWwith a category
labelci2f1;:::;Kgand a conﬁdence score si2[0;1]for
each object instance idetected in the video.
Evaluation Metric. Video instance segmentation is evaluated according to the metrics of average precision (AP) and
average recall (AR). Unlike in the image domain, these metrics are evaluated over the video sequence. Thus, to evaluate spatiotemporal consistency of the predicted mask sequences, the video Intersection over Union (IoU) between
a predicted object instance iand a ground truth object instancejis computed as:
IoU(i;j) =PT
t=1jMi(t)\~Mj(t)j
PT

CVPR 2020 Best Paper Nominee


IoU(i;j) =PT
t=1jMi(t)\~Mj(t)j
PT
t=1jMi(t)[~Mj(t)j(1)
where ~Mj(t)is the ground-truth segmentation of object j
in framet. To achieve a large IoU, a model must not only
accurately classify and segment object instances at a framelevel, but also reliably track them over the video sequence.
As in the COCO benchmark for image segmentation [25], the metrics of AP and AR are computed separatelyfor each object category, and then averaged over ten IoU
thresholds from 50% to95% at increments of 5%. Lastly,
the resulting AP and AR metrics are averaged over the category set, which yields the ﬁnal evaluation metric.
4. Mask Propagation
MaskProp takes a video Vof arbitrary length Las input and outputs video-level instance segmentation tracks
Mi, category labels ciand conﬁdence scores sifor all
objectsidetected in the video. In order to achieve this
goal, our method ﬁrst builds clip-level object instance tracks
Mi
tT:t+T2 R(2T+1)1HWfor each individual clip

CVPR 2020 Best Paper Nominee


Mi
tT:t+T2 R(2T+1)1HWfor each individual clip
VtT:t+T2R(2T+1)3HWof length (2T+ 1) in the
video, i.e., for t= 1;2;:::;L (clips at the beginning and
the end of the video will include fewer frames).
We want to use clips that are long enough to allow us
to jointly solve instance segmentation and tracking while
handling challenging cases of occlusion and motion blur.
At the same time, the clip should be short enough to allow
us to ﬁt it at high spatial resolution in the memory of a GPU.
The resulting clip-level instance masks Mi
tT:t+Tproduced densely for all overlapping clips t= 1;:::;L are
then aggregated to produce video-level instance masks Mi.
Our approach for clip-level instance segmentation is described in subsections 4.1 and 4.2. We also illustrate it in
Figure 2. The subsequent clip-level instance mask aggregation method is presented in subsection 4.3.
Frame t
Bbox...
ClsMask
Frame t+...     Mask  Propagation     CNN

CVPR 2020 Best Paper Nominee


Figure 3: We adapt Mask R-CNN [16] to video by adding
a mask propagation branch, for tracking object segmentation instances in video. Given a video clip centered around
framet, our system outputs a clip-level instance segmentation track as well as a classiﬁcation score and a bounding
box for each object instanced detected in frame t. For compactness, in this ﬁgure, we illustrate our system processing a
pair of frames but typically the propagation is applied from
the middle frame to all the other frames in the clip.
4.1. Video Mask R-CNN
Our video instance segmentation system is based on the
Mask R-CNN [16] model, which we adapt to video by
adding a mask propagation branch (See Figure 3). We train
our system with a multi-task loss Lt=Lcls
t+Lbox
t+
Lmask
t +Lprop
t wheretdenotes a time-step of a center
frame. We use identical loss terms Lcls
t;Lbox
t,Lmask
t as
in Mask R-CNN. The mask propagation loss is deﬁned as:
Lprop
t =~NtX
it+TX
t0=tT1sIoU (Mi
tT:t+T(t0);~Mi

CVPR 2020 Best Paper Nominee


t =~NtX
it+TX
t0=tT1sIoU (Mi
tT:t+T(t0);~Mi
tT:t+T(t0))(2)
whereMi
tT:t+T(t0)2[0;1]is the segmentation at time
t0for an instance ipredicted from a clip centered at tand
~Mi
tT:t+T(t0)is the corresponding ground truth mask at
timet0.~Ntis the number of ground truth object instances
in framet, andsIoU is deﬁned as:
sIoU (A;B) =P
pA(p)B(p)P
pA(p) +B(p)A(p)B(p)(3)
where the summations in numerator and denominator are
performed over every pixel location p. The loss above is a
soft IoU loss, which we observed to work slightly better
than the standard cross entropy loss for our task.
4.2. Mask Propagation Branch
Overview. Our main technical contribution is the design of
a mask propagation branch, that allows our method to track

CVPR 2020 Best Paper Nominee


4.2. Mask Propagation Branch
Overview. Our main technical contribution is the design of
a mask propagation branch, that allows our method to track
object instances. Given a video clip VtT:t+Tcenteredat framet, our system outputs clip-level instance masks
Mi
tT:t+Tfor each predicted object instance iin framet.
Our mask propagation branch can be described in three
high-level steps: 1) instance-speciﬁc feature computation,
2) temporal propagation of instance features, and 3) propagated instance segmentation. We will now describe each of
these steps in more detail. We introduce our mask propagation with the example of propagating object instance masks
from frametto framet+where2[T:T].
Computing Instance Speciﬁc Features. The mask
branch of our model predicts frame-level instance masks
Mi
t2R1H0W0from single frame inputs. We then
use these frame-level instance masks to compute instancespeciﬁc features for frame t. Speciﬁcally, for each object instancei, we compute an element-wise product between Mi
t
and the feature tensor from the backbone network ft. This
then yields a set of new feature tensors fi

CVPR 2020 Best Paper Nominee


t
and the feature tensor from the backbone network ft. This
then yields a set of new feature tensors fi
t2RCH0W0,
wherei= 1;::;Nt, andNtis the number of object instances
detected in frame t. In other words, for each object instance
i, we are zeroing out the feature values in ftthat correspond
to pixels not belonging to that object instance.
Temporally Propagating Instance Features. Given
frame-level features ft;ft+2RCH0W0and instancespeciﬁc feature tensor fi
t, our method generates a propagated instance feature tensor gi
t;t+. Intuitively, gi
t;t+represents the features predicted by our model for object instanceiin framet+from an instance-speciﬁc feature tensorfi
t. The tensor gi
t;t+is generated by warping features
fi
tusing the alignment computed from frame-level features
ftandft+. We implement the propagation mechanism via
a deformable convolution [12], which has previously been
used for aligning features computed from separate frames

CVPR 2020 Best Paper Nominee


ftandft+. We implement the propagation mechanism via
a deformable convolution [12], which has previously been
used for aligning features computed from separate frames
of a video [5, 3]. Speciﬁcally, we compute the elementwise difference of tensors ft;ft+and feed it through a
simple residual block [17], which predicts motion offsets
ot;t+2R2k2H0W0. These offsets contain (x;y)sampling locations for each entry of a kkdeformable convolution kernel [12]. The propagation step takes as inputs 1)
the offsetsot;t+and 2) the instance feature tensor fi
t, and
then applies deformable convolution to output the propagated instance feature tensor gi
t;t+for each instance i. We
use subscript t;t+to denote the propagated instance feature because, although gis obtained by propagating the feature tensorfi
t, the offset computation uses both frame tand
framet+. We stress that no explicit ground truth alignment is available between frames. The deformable convolutional kernels are supervised implicitly by optimizing Eq. 2.
Segmenting Propagated Instances. Lastly, we use our
propagated feature map gi

CVPR 2020 Best Paper Nominee


Segmenting Propagated Instances. Lastly, we use our
propagated feature map gi
t;t+for predicting a corresponding object instance mask in frame t+. To do this we
ﬁrst, construct a new feature tensor i
t;t+=gi
t;t++ft+.
The addition effectively overimposes the tensor gi
t;t+predicted from time tfor object instance iin framet+, with
Tensor t

CVPR 2020 Best Paper Nominee


Propagated Instance Segmentation (time t+  )<latexit

CVPR 2020 Best Paper Nominee


2) Instance Feature Propagation (from time t to time t+  )<latexit

CVPR 2020 Best Paper Nominee


Instance Feature Computation (time t)Instancej

CVPR 2020 Best Paper Nominee


4: An illustration of the 3 steps implemented by our mask propagation branch: 1)For every detected instance in frame

CVPR 2020 Best Paper Nominee


t, we compute an instance-speciﬁc feature tensor via element-wise multiplication between tensor tand the given frame-level
instance mask at frame t.2)Next, the element-wise difference of the feature tensors associated with frames tandt+is
used to predict motion offsets between frames tandt+. The resulting offsets are used to propagate the instance-speciﬁc
tensors from time tto timet+via deformable convolution. The propagated tensors represent instance-speciﬁc features
predicted for time t+using the tensors computed at time t.3)Lastly, we add the propagated instance feature tensors to the
tensor effectively computed at t+. A convolutional layer applied to these tensors predicts instance masks in frame t+.
The pixels that do not belong to any object instances are zeroed out using an instance-agnostic attention map.
the tensorft+effectively computed from the frame at time
t+. If the object instance prediction is consistent with the
feature computation, the feature tensors will be aligned and
thus, they will reinforce each other in the predicted region.
Finally, the resulting feature tensors i
t;t+are fed into
a11convolution layer that outputs instance masks for

CVPR 2020 Best Paper Nominee


Finally, the resulting feature tensors i
t;t+are fed into
a11convolution layer that outputs instance masks for
each object instance iin framet+. The masks are normalized with the softmax nonlinearity across all Ntinstances. To zero-out pixels that do not belong to any object
instance, we use a single 33convolution that computes
an instance-agnostic attention map At+from feature tensorft+. We then multiply At+with each of our predicted
instance masks. A detailed illustration of our mask propagation branch is presented in Figure 4.
4.3. Video-Level Segmentation Instances
Given a video of length L, our goal is to produce videolevel segmentation instances Mi2 RLHW. Conceptually, this requires linking clip-level instance tracks
Mi
tT:t+TandMj
t0T:t0+Twheniandjrepresent the same
object instance, i.e., when the instances are matching. We
achieve this by assigning a video-level instance ID to each
of our predicted clip-level instance tracks. Matching instance tracks are assigned the same video-level instance ID.
Matching Clip-Level Instance Tracks. Consider a pair of
clip-level instance tracks Mi

CVPR 2020 Best Paper Nominee


Matching Clip-Level Instance Tracks. Consider a pair of
clip-level instance tracks Mi
tT:t+TandMj
t0T:t0+T, that
are centered around frames tandt0respectively. They overlap in time ifjtt0j<2T+ 1. Let us denote their overlapping time interval as \t;t0. Given two overlapping in-stance tracks, we can check if they match by comparing
their predicted instance masks in the overlapping frames.
If the masks of the two instance tracks align well, then they
are likely to encode the same object instance. Otherwise,
each track represents a different object instance. We compute a matching score mi;j
t;t0between two clip-level instance
tracks using our previously deﬁned soft IoU metric as:
mi;j
t;t0=1
j\t;t0jX
~t2\t;t0sIoU (Mi
tT:t+T(~t); Mj
t0T:t0+T(~t))(4)
Video-Level Instance ID Assignment. We denote withY
the set of video-level instance IDs. The set Yis built incrementally by matching clip-level instance tracks Mi
tT:t+T

CVPR 2020 Best Paper Nominee


the set of video-level instance IDs. The set Yis built incrementally by matching clip-level instance tracks Mi
tT:t+T
in order from time t= 1 to timet=L. Initially, we set
Y=f1;2;:::;N 1gwhereN1is the number of object instances detected at time t= 1 (i.e., in the ﬁrst clip of the
video). Let yi
t2Y denote the video-level instance ID assigned to clip-level instance track Mi
tT:t+T. As we move
forward in time t>1, the video-level instance ID yi
tis assigned by matching clip-level instance Mi
tT:t+Tto all previously processed instance tracks Mj
t0T:t0+Tthat overlap
with this clip, i.e., such that \t;t06=;. For each video-level
instance IDy2Y already in the ID set, we compute a score
qi
t(y)capturing how well Mi
tT:t+Tmatches the tracks that
have been already assigned video-level ID y:
qi
t(y) =P
t0s.t.\t;t06=;PNt0
j=11fyj

CVPR 2020 Best Paper Nominee


qi
t(y) =P
t0s.t.\t;t06=;PNt0
j=11fyj
t0=ygmi;j
t;t0
P
t0s.t.\t;t06=;PNt0
j=11fyj
t0=yg(5)
time ttime t+5time t+10time t+15time t-15time t-10time t-5
Figure 5: An illustration of instance-speciﬁc features propagated from frame tto other frames in the given video clip. Here,
we visualize propagated activations from one randomly selected feature channel. The activations in the two rows correspond
to two different object instances detected at time t. Our visualizations suggest that MaskProp reliably propagates features
that are speciﬁc to each instance even when instances appear next to each other, and despite the changes in shape, pose and
the nuisances effects of deformation and occlusion.
where 1fyj
t0=ygis an indicator function that is equal
to1whenyj
t0=y, and 0otherwise.Nt0is the number
of detected instances at time t0, andmi;j
t;t0is the previously
deﬁned matching score between instance tracks Mi

CVPR 2020 Best Paper Nominee


of detected instances at time t0, andmi;j
t;t0is the previously
deﬁned matching score between instance tracks Mi
tT:t+T
andMj
t0T:t0+T. The score qi
t(y)is effectively an average
of the soft IoU computed between instance track Mi
tT:t+T,
and all its overlapping instance tracks that have been assigned video-level instance ID y.
Letq= maxy2Yqi
t(y)be the maximum score obtained
by considering all possible video-level instance IDs y2Y.
Ifqis greater than a certain threshold, the current instance track Mi
tT:t+Tis assigned the video-level instance
IDy= arg maxy2Yqi
t(y). Otherwise, the clip-level track
does not match any of the current video-level instances. In
such case we create a new video-level instance ID and assign it to the the clip-level track while also expanding the
setY, i.e.,yi
t=jYj+ 1andY=Y[fjYj + 1g.
Finally, for each video-level instance ID y2 Y , we
generate the ﬁnal sequence of segmentation instance masks

CVPR 2020 Best Paper Nominee


Finally, for each video-level instance ID y2 Y , we
generate the ﬁnal sequence of segmentation instance masks
My2RLHWas:
My(t) =(
Mi
tT:t+T(t)ifyi
t=y
0 otherwise .4.4. Implementation Details
Backbone Network. As our backbone we use a Spatiotemporal Sampling Network [5] based on a Deformable
ResNeXt-101-64x4d [12, 41] with a feature pyramid network (FPN) [24] attached on top of it.
Detection Network. For detection, we use a Hybrid Task
Cascade Network [7] with a 3stage cascade.
Mask Propagation Branch. The residual block in the
mask propagation branch consists of two 33convolutions with 128output channels each. The instance feature
propagation is applied to the FPN feature map with the second largest spatial resolution. To propagate instance features from one frame to another, we predict 9 (x;y)offsets
for every pixel, which are then used as input to a 33deformable convolutional layer with 256channels. To capture
motion at different scales, we use three levels of dilated deformable convolutions with dilation rates 3;6;12as in [3].

CVPR 2020 Best Paper Nominee


motion at different scales, we use three levels of dilated deformable convolutions with dilation rates 3;6;12as in [3].
High-Resolution Mask Reﬁnement. Predicting masks
from RoI features typically leads to low resolution predictions. We address this issue via a high-resolution mask reﬁnement step. Given the center of a detected bounding box,
we crop a 384384patch around the object, preserving
the original aspect ratio. We then feed the RGB patch and
the predicted low-resolution mask through 3residual blocks
each with 128channels to obtain a high-resolution mask.
Method Pre-training Data mAP AP@75 AR@1 AR@10
DeepSORTz[39] Imagenet [34], COCO [25] 26.1 26.1 27.8 31.3
FEELVOSz[37] Imagenet [34], COCO [25] 26.9 29.7 29.9 33.4
OSMNz[43] Imagenet [34], COCO [25] 27.5 29.1 28.6 33.1
MaskTrack R-CNNz[42] Imagenet [34], COCO [25] 30.3 32.6 31.0 35.5

CVPR 2020 Best Paper Nominee


MaskTrack R-CNNz[42] Imagenet [34], COCO [25] 30.3 32.6 31.0 35.5
MaskTrack R-CNNImagenet [34], COCO [25] 36.9 40.2 34.3 42.9
EnsembleVIS [28] Imagenet [34], COCO [25], Instagram [30], OpenImages [23] 44.8 48.9 42.7 51.7
MaskProp Imagenet [34], COCO [25] 46.6 51.2 44.0 52.6
MaskProp Imagenet [34], COCO [25], OpenImages [23] 50.0 55.9 44.6 54.5
Table 2: The results of video instance segmentation on the YouTube-VIS [42] validation dataset. We evaluate the performance
of each method according to mean average precision (mAP), average precision at 75% IoU threshold (AP@75), and average
recall given top 1(AR@1) and top 10(AR@10) detections. The baselines denoted withzwere implemented by the authors
in [42], whereas the methods marked withwere implemented by us, and use the same backbone and detection networks as
our approach. Despite its simplicity, MaskProp outperforms all prior video instance segmentation methods by a large margin.

CVPR 2020 Best Paper Nominee


our approach. Despite its simplicity, MaskProp outperforms all prior video instance segmentation methods by a large margin.
Scoring Video-Level Sequences. Each video-level sequence contains a list of classiﬁcation scores and predicted
object labels. To assign a conﬁdence score to each videolevel sequence, we average classiﬁcation scores associated
with that sequence (separately for each object category).
Training. We use a similar training setup as in [7]. The loss
weights for each of the three cascade stages are set to 1;0:5;
and0:25respectively. The loss weight for the semantic segmentation branch is set to 0:1. We train our model on pairs
of frames, where the second frame in a pair is randomly selected with a time gap 2[25;25]relative to ﬁrst frame.
We use a multi-scale training approach implemented by resizing the shorter side of the frame randomly between 400
and800pixels. Our model is trained in a distributed setting using 64GPUs, each GPU holding a single clip. The
training is done for 20epochs with an initial learning rate
of0:008, which is decreased by 10at16and19epochs.
We initialize our model with a Mask R-CNN pretrained on

CVPR 2020 Best Paper Nominee


of0:008, which is decreased by 10at16and19epochs.
We initialize our model with a Mask R-CNN pretrained on
COCO for the instance segmentation. The hyperparameters
of RPN and FPN are the same as in [7].
Inference. During testing, we run the bounding box prediction branch on 1000 proposals, apply non-maximum suppression, and use boxes with a score higher than 0:1as input to the mask prediction and mask propagation branches.
During inference, our MaskProp is applied to video clips
consisting of 13frames.
5. Experimental Results
In this section, we evaluate MaskProp for video instance
segmentation on YouTube-VIS [42], which contains 2;238
training, 302validation, and 343test videos. Each video
is annotated with per-pixel segmentation, category, and instance labels. The dataset contains 40object categories.
Since the evaluation on the test set is currently closed, we
perform our evaluations on the validation set.
5.1. Quantitative Results
Video instance segmentation is a very recent task [42],
and thus, there are only a few established baselines that we
7 9 11 13 15 17 19 21

CVPR 2020 Best Paper Nominee


Video instance segmentation is a very recent task [42],
and thus, there are only a few established baselines that we
7 9 11 13 15 17 19 21
Instance Track Clip Length43.54444.54545.54646.547Accuracy (Video mAP)Figure 6: We plot video mAP as a function of an instance
track clip length (denoted as 2T+ 1in the paper). Based on
these results, we observe that optimal video instance segmentation performance is achieved when we propagate instance masks to T= 6previous and subsequent frames.
can compare our work to. We include in our comparison
MaskTrack R-CNN [42], and the EnsembleVIS method,
which won the ICCV 2019 video instance segmentation
challenge [28]. Additionally, to make the comparison with
MaskTrack R-CNN more fair, we reimplement it using the
same backbone and detection networks as our MaskProp
(see MaskTrack R-CNN* in Table 2).
We present our quantitative results in Table 2, where we
assess each method according to 1) mean video average precision (mAP), 2) video average precision at IoU threshold
of75%, and 3) average recall given 1and10highest scored
instances per video. From these results, we observe that
our MaskProp outperforms all the other baselines according

CVPR 2020 Best Paper Nominee


of75%, and 3) average recall given 1and10highest scored
instances per video. From these results, we observe that
our MaskProp outperforms all the other baselines according
to all four evaluation metrics, thus achieving state-of-theart results in video instance segmentation on YouTube-VIS.
It can be noted that we outperform EnsembleVIS [28] by
1:8mAP even though our approach is much simpler and
even if it uses orders of magnitude less labeled data for
pre-training. Furthermore, compared to the MaskTrack RCNN, our method achieves a 16:3improvement in mAP. We
also note that our implementation of MaskTrack [42] signiﬁcantly improves upon the original work, but it is still 9:7
Method mAP AP@75
FlowNet2 Propagation 31.4 33.6
MaskTrack R-CNN36.9 40.2
MaskProp 46.6 51.2
Table 3: Here, we study the effectiveness of our mask propagation branch. If we replace it with the FlowNet2 propagation scheme, where masks are propagated using the optical
ﬂow predicted by a FlowNet2 network [20], the accuracy
drops from 46:6mAP to 31:4mAP. Similarly, if we replace
our mask propagation branch with the tracking branch from

CVPR 2020 Best Paper Nominee


drops from 46:6mAP to 31:4mAP. Similarly, if we replace
our mask propagation branch with the tracking branch from
MaskTrack R-CNN, the accuracy drops to 36:9mAP. Note
that all of these baselines are implemented using the same
backbone and detection networks.
worse in mAP compared to our model. Lastly, we note that
additionally pre-training MaskProp on the OpenImages [23]
dataset as was done in [28] further boosts our performance
to50:0mAP.
5.2. Ablation Experiments
Mask Propagation Branch. To investigate the effectiveness of our mask propagation branch, we compare our
method with a FlowNet2 [20] propagation baseline. For
this baseline, we use exactly the same setup as for our
MaskProp, except that instance masks are propagated using the optical ﬂow predicted by a FlowNet2 network [20]
rather than our proposed mask propagation scheme. For a
more complete comparison, we also include the MaskTrack
R-CNNfrom Table 2, which uses the originally proposed
tracking branch [42], but is implemented using the same
backbone and detection networks as our MaskProp.
These baselines allow us to directly compare the effectiveness of our mask propagation scheme versus the propagation mechanisms employed by FlowNet2 and MaskTrack

CVPR 2020 Best Paper Nominee


backbone and detection networks as our MaskProp.
These baselines allow us to directly compare the effectiveness of our mask propagation scheme versus the propagation mechanisms employed by FlowNet2 and MaskTrack
R-CNN [42] methods. The results in Table 3 show that
MaskProp outperforms these baselines by a large margin.
Instance Track Clip Length. Due to occlusions, object instances in video may not be visible in some frames. If there
are occlusions separated by 2T0+ 2time-steps, we can use
T >T0to predict longer clip-level instance tracks. In Figure 6, we study video instance segmentation performance as
a function of instance track clip length (denoted as 2T+ 1
in the paper). Our results indicate that the best accuracy is
achieved when we use a clip length of 13, meaning that we
propagate instances to 6previous and 6subsequent frames.
High-Resolution Mask Reﬁnement. We also study the impact of our high-resolution mask reﬁnement, described in
Subsection 4.4. We report that removing this reﬁnement
causes a drop of 1:9%in video instance segmentation mAP.
Importance of Frame-Level Instance Masks. As described in our main draft, we use frame-level instance masks

CVPR 2020 Best Paper Nominee


Importance of Frame-Level Instance Masks. As described in our main draft, we use frame-level instance masks
for instance-speciﬁc feature computation. To investigate theBackbone Network mAP AP@75
ResNet-50 40.0 42.9
ResNet-101 42.5 45.6
ResNeXt-101-64x4d 44.3 48.3
STSN [5]-ResNeXt-101-64x4d 46.6 51.2
Table 4: We study the effect of frame-level instance masks
to our system’s performance. We evaluate our method’s
accuracy when using instance masks obtained from Mask
R-CNN with several different backbones. Our results indicate that frame-level instance masks obtained from stronger
models lead to better video instance segmentation results.
contribution of these masks to the performance of our system, we experiment with masks obtained from several different Mask R-CNN models. In Table 4, we present our
results for this ablation. Our results indicate that framelevel instance masks obtained from stronger models allow
us to achieve better video instance segmentation performance. Thus, we expect that future improvements in image
instance segmentation will further beneﬁt our method.
5.3. Qualitative Results
In Figure 7, we compare our predicted clip-level instance

CVPR 2020 Best Paper Nominee


instance segmentation will further beneﬁt our method.
5.3. Qualitative Results
In Figure 7, we compare our predicted clip-level instance
tracks (last row of predictions for each clip) with the MaskTrack R-CNN predictions (ﬁrst row of predictions). We
use different colors to represent different object instances.
Our qualitative results suggest that our MaskProp produces
more robust and temporally coherent instance tracks than
MaskTrack R-CNN. Such differences in performance are
especially noticeable when a video contains large object
motion, occlusions, or overlapping objects.
In Figure 5, we also visualize instance-speciﬁc features
that are propagated from frame tto other frames in the
given video clip for two different object instances detected
in framet. Here, we show activations from a randomly selected feature channel. Based on these results, we observe
that our MaskProp reliably propagates features that are speciﬁc to each instance despite motion blur, object deformations and large variations in object appearance.
6. Conclusion
In this work, we introduced MaskProp, a novel architecture for video instance segmentation. Our method is conceptually simple, it does not require large amounts of labeled data for pre-training, and it produces state-of-the-art

CVPR 2020 Best Paper Nominee


results on YouTube-VIS dataset. In future, we plan to extend MaskProp to scenarios where only bounding box annotations are available. We are also interested in applying our
method to problems such as pose estimation and tracking.
Video ClipMaskPropMaskTrack      R-CNN
Video ClipMaskPropMaskTrack      R-CNN
Video Clip
MaskProp
MaskTrack      R-CNNVideo Clip
MaskPropMaskTrack      R-CNN
Figure 7: We compare our video instance segmentation results with MaskTrack R-CNN [42] predictions. Different object
instances are encoded with different colors. The ﬁrst row for each video shows the original frames. The second row illustrates
the mask predictions of MaskTrack R-CNN and the third row those obtained with our MaskProp. Compared to MaskTrack
R-CNN, our MaskProp tracks object instances more robustly even when they are occluded or overlap with each other.
References
[1] Anurag Arnab and Philip H. S. Torr. Pixelwise instance segmentation with a dynamically instantiated network. In 2017
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,
pages 879–888, 2017. 2
[2] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In 2017 IEEE Conference on Computer

CVPR 2020 Best Paper Nominee


pages 879–888, 2017. 2
[2] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) , pages 2858–2866,
July 2017. 2
[3] Gedas Bertasius, Christoph Feichtenhofer, Du Tran, Jianbo
Shi, and Lorenzo Torresani. Learning temporal pose estimation from sparsely labeled videos. In Advances in Neural
Information Processing Systems 33 . 2019. 4, 6
[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural ﬁelds. In The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 1
[5] Gedas Bertasius, Lorenzo Torresani, and Jianbo Shi. Object
detection in video with spatiotemporal sampling networks.
InECCV (12) , volume 11216 of Lecture Notes in Computer
Science , pages 342–357. Springer, 2018. 2, 4, 6, 8
[6] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset,

CVPR 2020 Best Paper Nominee


[6] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset,
Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. Oneshot video object segmentation. In Computer Vision and Pattern Recognition (CVPR) , 2017. 2
[7] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao
Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi,
Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid
task cascade for instance segmentation. In IEEE Conference
on Computer Vision and Pattern Recognition, CVPR 2019,
Long Beach, CA, USA, June 16-20, 2019 , pages 4974–4983.
Computer Vision Foundation / IEEE, 2019. 6, 7
[8] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,
Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal.
Mach. Intell. , 40(4):834–848, 2018. 1

CVPR 2020 Best Paper Nominee


Mach. Intell. , 40(4):834–848, 2018. 1
[9] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian
Schroff, and Hartwig Adam. Encoder-decoder with atrous
separable convolution for semantic image segmentation. In
Computer Vision - ECCV 2018 - 15th European Conference,
Munich, Germany, September 8-14, 2018, Proceedings, Part
VII, pages 833–851, 2018. 2
[10] Yuhua Chen, Jordi Pont-Tuset, Alberto Montes, and Luc Van
Gool. Blazingly fast video object segmentation with pixelwise metric learning. In CVPR , pages 1189–1198. IEEE
Computer Society, 2018. 2
[11] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In
CVPR , 2016. 2
[12] J. Dai, H. Qi, Y . Xiong, Y . Li, G. Zhang, H. Hu, and Y . Wei.
Deformable convolutional networks. In 2017 IEEE International Conference on Computer Vision (ICCV) , volume 00,

CVPR 2020 Best Paper Nominee


Deformable convolutional networks. In 2017 IEEE International Conference on Computer Vision (ICCV) , volume 00,
pages 764–773, Oct. 2017. 4, 6[13] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.
Detect to track and track to detect. In International Conference on Computer Vision (ICCV) , 2017. 2
[14] Ross Girshick. Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV) , 2015. 1
[15] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra
Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , 2014. 1
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the International
Conference on Computer Vision (ICCV) , 2017. 1, 2, 4
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. 2016 IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , pages 770–778, 2016. 1, 4

CVPR 2020 Best Paper Nominee


Deep residual learning for image recognition. 2016 IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , pages 770–778, 2016. 1, 4
[18] Alexander Hermans*, Lucas Beyer*, and Bastian Leibe. In
Defense of the Triplet Loss for Person Re-Identiﬁcation.
arXiv preprint arXiv:1703.07737 , 2017. 2
[19] Alba Herrera-Palacio, Carles Ventura, and Xavier Giro-i Nieto. Video object linguistic grounding. In 1st International
Workshop on Multimodal Understanding and Learning for
Embodied Applications , MULEA ’19, pages 49–51, New
York, NY , USA, 2019. ACM. 2
[20] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper,
Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical ﬂow estimation with deep networks. CoRR ,
abs/1612.01925, 2016. 8
[21] Suyog Dutt Jain, Bo Xiong, and Kristen Grauman. Fusionseg: Learning to combine motion and appearance for fully
automatic segmentation of generic objects in videos. In 2017

CVPR 2020 Best Paper Nominee


automatic segmentation of generic objects in videos. In 2017
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,
pages 2117–2126, 2017. 2
[22] Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savchynskyy, and Carsten Rother. InstanceCut: from
edges to instances with multicut. In CVPR , 2017. 2
[23] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan
Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari.
The open images dataset v4: Uniﬁed image classiﬁcation,
object detection, and visual relationship detection at scale,
2018. 2, 7, 8
[24] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He,
Bharath Hariharan, and Serge Belongie. Feature pyramid
networks for object detection. In CVPR , 2017. 6
[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,

CVPR 2020 Best Paper Nominee


networks for object detection. In CVPR , 2017. 6
[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence
Zitnick. Microsoft coco: Common objects in context. In
European Conference on Computer Vision (ECCV) , Zürich,
September 2014. 2, 3, 7
[26] S. Liu, J. Jia, S. Fidler, and R. Urtasun. Sgn: Sequential
grouping networks for instance segmentation. In 2017 IEEE
International Conference on Computer Vision (ICCV) , pages
3516–3524, Oct 2017. 2
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
convolutional networks for semantic segmentation. In
CVPR , 2015. 1
[28] Jonathon Luiten, Philip H.S. Torr, and Bastian Leibe. Video
instance segmentation 2019: A winning approach for combined detection, segmentation, classiﬁcation and tracking.
In2019 IEEE International Conference on Computer Vision
Workshops, ICCV Workshops 2019, Seoul, South Korea, October 27-November 2, 2019 . IEEE, 2019. 1, 2, 7, 8

CVPR 2020 Best Paper Nominee


Workshops, ICCV Workshops 2019, Seoul, South Korea, October 27-November 2, 2019 . IEEE, 2019. 1, 2, 7, 8
[29] Jonathon Luiten, Paul V oigtlaender, and Bastian Leibe. Premvos: Proposal-generation, reﬁnement and merging for
video object segmentation. In Asian Conference on Computer Vision , 2018. 2
[30] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan,
Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe,
and Laurens van der Maaten. Exploring the limits of weakly
supervised pretraining. In ECCV , 2018. 2, 7
[31] Aljoša Ošep, Paul V oigtlaender, Jonathon Luiten, Stefan
Breuers, and Bastian Leibe. Large-scale object mining for
object discovery from unlabeled video. ICRA , 2019. 2
[32] F. Perazzi, A. Khoreva, R. Benenson, B. Schiele, and
A.Sorkine-Hornung. Learning video object segmentation
from static images. In Computer Vision and Pattern Recognition , 2017. 2

CVPR 2020 Best Paper Nominee


A.Sorkine-Hornung. Learning video object segmentation
from static images. In Computer Vision and Pattern Recognition , 2017. 2
[33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster R-CNN: Towards real-time object detection with region proposal networks. In Neural Information Processing
Systems (NIPS) , 2015. 1
[34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy,
Aditya Khosla, Michael Bernstein, Alexander C. Berg, and
Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575 , 2014. 2, 7
[35] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.
PWC-Net: CNNs for optical ﬂow using pyramid, warping,
and cost volume. 2018. 2
[36] P. Tokmakov, K. Alahari, and C. Schmid. Learning video
object segmentation with visual memory. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages
4491–4500, Oct 2017. 2

CVPR 2020 Best Paper Nominee


object segmentation with visual memory. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages
4491–4500, Oct 2017. 2
[37] Paul V oigtlaender, Yuning Chai, Florian Schroff, Hartwig
Adam, Bastian Leibe, and Liang-Chieh Chen. Feelvos: Fast
end-to-end embedding learning for video object segmentation. In CVPR , 2019. 2, 7
[38] Paul V oigtlaender, Michael Krause, Aljosa Osep, Jonathon
Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger,
and Bastian Leibe. Mots: Multi-object tracking and segmentation. In Proceedings IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR) , June 2019. 2
[39] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple
online and realtime tracking with a deep association metric.
CoRR , abs/1703.07402, 2017. 7
[40] Fanyi Xiao and Yong Jae Lee. Video object detection with an
aligned spatial-temporal memory. In European Conference
on Computer Vision (ECCV) , 2018. 2

CVPR 2020 Best Paper Nominee


[40] Fanyi Xiao and Yong Jae Lee. Video object detection with an
aligned spatial-temporal memory. In European Conference
on Computer Vision (ECCV) , 2018. 2
[41] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and
Kaiming He. Aggregated residual transformations for deep
neural networks. In CVPR , 2017. 6[42] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV , 2019. 1, 2, 3, 7, 8, 9
[43] Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang,
and Aggelos K. Katsaggelos. Efﬁcient video object segmentation via network modulation. In 2018 IEEE Conference on
Computer Vision and Pattern Recognition, CVPR 2018, Salt
Lake City, UT, USA, June 18-22, 2018 , pages 6499–6507,
2018. 2, 7
[44] Jifeng Dai Xiangyang Ji Yi Li, Haozhi Qi and Yichen Wei.
Fully convolutional instance-aware semantic segmentation.
InCVPR , 2017. 2
[45] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang

CVPR 2020 Best Paper Nominee


InCVPR , 2017. 2
[45] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang
Wang, and Jiaya Jia. Pyramid scene parsing network. In 2017
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,
pages 6230–6239, 2017. 1
[46] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen
Wei. Flow-guided feature aggregation for video object detection. In International Conference on Computer Vision
(ICCV) , 2017. 2
[47] Idil Esen Zulﬁkar, Jonathon Luiten, and Bastian Leibe. Unovost: Unsupervised ofﬂine video object segmentation and
tracking for the 2019 unsupervised davis challenge. 2

CVPR 2020 Best Paper Nominee


Quantifying Attention Flow in Transformers
Samira Abnar
ILLC, University of Amsterdam
s.abnar@uva.nlWillem Zuidema
ILLC, University of Amsterdam
w.h.zuidema@uva.nl
Abstract
In the Transformer model, “self-attention”
combines information from attended embeddings into the representation of the focal embedding in the next layer. Thus, across layers of the Transformer, information originating
from different tokens gets increasingly mixed.
This makes attention weights unreliable as explanations probes. In this paper, we consider
the problem of quantifying this ﬂow of information through self-attention. We propose two
methods for approximating the attention to input tokens given attention weights, attention
rollout andattention ﬂow , as post hoc methods
when we use attention weights as the relative
relevance of the input tokens. We show that
these methods give complementary views on
the ﬂow of information, and compared to raw
attention, both yield higher correlations with
importance scores of input tokens obtained using an ablation method and input gradients.
1 Introduction
Attention (Bahdanau et al., 2015; Vaswani et al.,
2017) has become the key building block of neural sequence processing models, and visualizing
attention weights is the easiest and most popular

Summary:
In the Transformer model, "self-attention" combines information from attended
embeddings into the representation of the focal embedding in the next layer.
Thus, across layers of the Transformer, information originating from different
tokens gets increasingly mixed. This makes attention weights unreliable as
explanations probes. In this paper, we consider the problem of quantifying this
flow of information through self-attention. We propose two methods for
approximating the attention to input tokens given attention weights, attention
rollout and attention flow, as post hoc methods when we use attention weights
as the relative relevance of the input tokens. We show that these methods give
complementary views on the flow of information, and compared to raw attention,
both yield higher correlations with importance scores of input tokens obtained
using an ablation method and input gradients.


2017) has become the key building block of neural sequence processing models, and visualizing
attention weights is the easiest and most popular
approach to interpret a model’s decisions and to
gain insights about its internals (Vaswani et al.,
2017; Xu et al., 2015; Wang et al., 2016; Lee et al.,
2017; Dehghani et al., 2019; Rockt ¨aschel et al.,
2016; Chen and Ji, 2019; Coenen et al., 2019; Clark
et al., 2019). Although it is wrong to equate attention with explanation (Pruthi et al., 2019; Jain and
Wallace, 2019), it can offer plausible and meaningful interpretations (Wiegreffe and Pinter, 2019;
Vashishth et al., 2019; Vig, 2019). In this paper,
we focus on problems arising when we move to the
higher layers of a model, due to lack of token identiﬁability of the embeddings in higher layers (Brunner et al., 2020).We propose two simple but effective methods to
compute attention scores to input tokens (i.e., token
attention ) at each layer, by taking raw attentions
(i.e., embedding attention ) of that layer as well as
those from the precedent layers. These methods


attention ) at each layer, by taking raw attentions
(i.e., embedding attention ) of that layer as well as
those from the precedent layers. These methods
are based on modelling the information ﬂow in the
network with a DAG (Directed Acyclic Graph), in
which the nodes are input tokens and hidden embeddings, edges are the attentions from the nodes
in each layer to those in the previous layer, and
the weights of the edges are the attention weights.
The ﬁrst method, attention rollout , assumes that
the identities of input tokens are linearly combined
through the layers based on the attention weights.
To adjust attention weights, it rolls out the weights
to capture the propagation of information from input tokens to intermediate hidden embeddings. The
second method, attention ﬂow , considers the attention graph as a ﬂow network. Using a maximum
ﬂow algorithm, it computes maximum ﬂow values,
from hidden embeddings (sources) to input tokens
(sinks). In both methods, we take the residual connection in the network into account to better model
the connections between input tokens and hidden
embedding. We show that compared to raw attention, the token attentions from attention rollout and
attention ﬂow have higher correlations with the importance scores obtained from input gradients as


embedding. We show that compared to raw attention, the token attentions from attention rollout and
attention ﬂow have higher correlations with the importance scores obtained from input gradients as
well as blank-out , an input ablation based attribution method. Furthermore, we visualize the token
attention weights and demonstrate that they are better approximations of how input tokens contribute
to a predicted output, compared to raw attention.
It is noteworthy that the techniques we propose
in this paper, are not toward making hidden embeddings more identiﬁable, or providing better attention weights for better performance, but a new set
of attention weights that take token identity problem into consideration and can serve as a better
diagnostic tool for visualization and debugging.arXiv:2005.00928v2  [cs.LG]  31 May 2020
(a) Embedding attentions
 (b) Attention rollout
 (c) Attention ﬂow
Figure 1: Visualisation of attention weights.
Figure 2: Raw Attention maps for the CLS token at
different layers.
2 Setups and Problem Statement
In our analysis, we focus on the verb number prediction task, i.e., predicting singularity or plurality
of a verb of a sentence, when the input is the sentence up to the verb position. We use the subjectverb agreement dataset (Linzen et al., 2016). This


of a verb of a sentence, when the input is the sentence up to the verb position. We use the subjectverb agreement dataset (Linzen et al., 2016). This
task and dataset are convenient choices, as they offer a clear hypothesis about what part of the input
is essential to get the right solution. For instance,
given “ the key to the cabinets ” as the input, we
know that attending to “key” helps the model predict singular as output while attending to “cabinets”
(anagreement attractor , with the opposite number)
is unhelpful.
We train a Transformer encoder, with GPT2 Transformer blocks as described in (Radford
et al., 2019; Wolf et al., 2019) (without masking).
The model has 6 layers, and 8 heads, with hidden/embedding size of 128. Similar to Bert (Devlin et al., 2019) we add a CLS token and use
its embedding in the ﬁnal layer as the input to
the classiﬁer. The accuracy of the model on the
subject-verb agreement task is 0:96. To facilitate
replication of our experiments we will make the
implementations of the models we use and algorithms we introduce publicly available at https:
//github.com/samiraabnar/attention_flow .


implementations of the models we use and algorithms we introduce publicly available at https:
//github.com/samiraabnar/attention_flow .
We start by visualizing raw attention in Figure 1a
(like Vig 2019). The example given here is correctly classiﬁed. Crucially, only in the ﬁrst couple
of layers, there are some distinctions in the attention patterns for different positions, while in higher
layers the attention weights are rather uniform. Fig-ure 2 (left) gives raw attention scores of the CLS
token over input tokens (x-axis) at different layers (y-axis), which similarly lack an interpretable
pattern.These observations reﬂect the fact that as
we go deeper into the model, the embeddings are
more contextualized and may all carry similar information. This underscores the need to track down
attention weights all the way back to the input layer
and is in line with ﬁndings of Serrano and Smith
(2019), who show that attention weights do not
necessarily correspond to the relative importance
of input tokens.
To quantify the usefulness of raw attention
weights, and the two alternatives that we consider
in the next section, besides input gradients, we
employ an input ablation method, blank-out , to estimate an importance score for each input token.


in the next section, besides input gradients, we
employ an input ablation method, blank-out , to estimate an importance score for each input token.
Blank-out replaces each token in the input, one
by one, with UNK and measures how much it affects the predicted probability of the correct class.
We compute the Spearman’s rank correlation coefﬁcient between the attention weights of the CLS
embedding in the ﬁnal layer and the importance
scores from blank-out. As shown in the ﬁrst row
of Table 1, the correlation between raw attention
weights of the CLS token and blank-out scores is
rather low, except for the ﬁrst layer. As we can see
in Table 2 this is also the case when we compute
the correlations with input gradients.
L1 L2 L3 L4 L5 L6
Raw 0.690.27 0.100.43 -0.110.49 -0.090.52 0.200.45 0.290.39
Rollout 0.320.26 0.380.27 0.510.26 0.620.26 0.700.25 0.710.24


Flow 0.320.26 0.440.29 0.700.25 0.700.22 0.710.22 0.700.22
Table 1: SpearmanR correlation of attention based importance with blank-out scores for 2000 samples from
the test set for the verb number prediction model.
3 Attention Rollout and Attention Flow
Attention rollout and attention ﬂow recursively
compute the token attentions in each layer of a
L1 L2 L3 L4 L5 L6
Raw 0.530.33 0.160.38 -0.060.42 0.000.47 0.240.40 0.460.35
Rollout 0.220.31 0.270.32 0.390.32 0.470.32 0.530.32 0.540.31
Flow 0.220.31 0.310.34 0.540.32 0.610.28 0.600.28 0.610.28
Table 2: SpearmanR correlation of attention based importance with input gradients for 2000 samples from
the test set for the verb number prediction model.


Table 2: SpearmanR correlation of attention based importance with input gradients for 2000 samples from
the test set for the verb number prediction model.
given model given the embedding attentions as input. They differ in the assumptions they make
about how attention weights in lower layers affect
the ﬂow of information to the higher layers and
whether to compute the token attentions relative to
each other or independently.
To compute how information propagates from
the input layer to the embeddings in higher layers, it is crucial to take the residual connections
in the model into account as well as the attention
weights. In a Transformer block, both self-attention
and feed-forward networks are wrapped by residual connections, i.e., the input to these modules is
added to their output. When we only use attention
weights to approximate the ﬂow of information in
Transformers, we ignore the residual connections.
But these connections play a signiﬁcant role in
tying corresponding positions in different layers.
Hence, to compute attention rollout and attention
ﬂow, we augment the attention graph with extra
weights to represent residual connections. Given
the attention module with residual connection, we
compute values in layer l+1asVl+1=Vl+WattVl,
where Wattis the attention matrix. Thus, we have


compute values in layer l+1asVl+1=Vl+WattVl,
where Wattis the attention matrix. Thus, we have
Vl+1= (Watt+I)Vl. So, to account for residual
connections, we add an identity matrix to the attention matrix and re-normalize the weights. This
results in A= 0:5Watt+ 0:5I, where Ais the raw
attention updated by residual connections.
Furthermore, analyzing individual heads requires accounting for mixing of information between heads through a position-wise feed-forward
network in Transformer block. Using attention rollout and attention ﬂow, it is also possible to analyze
each head separately. We explain in more details
in Appendix A.1. However, in our analysis in this
paper, for simplicity, we average the attention at
each layer over all heads.
Attention rollout Attention rollout is an intuitive
way of tracking down the information propagated
from the input layer to the embeddings in the higher
layers. Given a Transformer with Llayers, we want
to compute the attention from all positions in layerlito all positions in layer lj, where j < i . In the
attention graph, a path from node vat position k
inli, to node uat position minlj, is a series of


attention graph, a path from node vat position k
inli, to node uat position minlj, is a series of
edges that connect these two nodes. If we look
at the weight of each edge as the proportion of
information transferred between two nodes, we
can compute how much of the information at v
is propagated to uthrough a particular path by
multiplying the weights of all edges in that path.
Since there may be more than one path between
two nodes in the attention graph, to compute the
total amount of information propagated from vtou,
we sum over all possible paths between these two
nodes. At the implementation level, to compute the
attentions from litolj, we recursively multiply the
attention weights matrices in all the layers below.
~A(li) =
A(li)~A(li1)ifi > j
A(li) ifi=j(1)
In this equation, ~Ais attention rollout, Ais raw attention and the multiplication operation is a matrix
multiplication. With this formulation, to compute
input attention we set j= 0.
Attention ﬂow In graph theory, a ﬂow network
is a directed graph with a “capacity” associated


input attention we set j= 0.
Attention ﬂow In graph theory, a ﬂow network
is a directed graph with a “capacity” associated
with each edge. Formally, given G= (V; E)is a
graph, where Vis the set of nodes, and Eis the set
of edges in G;C=fcuv2Rj8u; vwhere eu;v2
E^u6=vgdenotes the capacities of the edges and
s; t2Vare the source and target (sink) nodes respectively; ﬂow is a mapping of edges to real numbers,f:E!R, that satisﬁes two conditions: (a)
capacity constraint : for each edge the ﬂow value
should not exceed its capacity, jfuvcuvj; (b)
ﬂow conservation : for all nodes except sandtthe
input ﬂow should be equal to output ﬂow –sum
of the ﬂow of outgoing edges should be equal to
sum of the ﬂow of incoming edges. Given a ﬂow
network, a maximum ﬂow algorithm ﬁnds a ﬂow
which has the maximum possible value between s
andt(Cormen et al., 2009).


network, a maximum ﬂow algorithm ﬁnds a ﬂow
which has the maximum possible value between s
andt(Cormen et al., 2009).
Treating the attention graph as a ﬂow network,
where the capacities of the edges are attention
weights, using any maximum ﬂow algorithm, we
can compute the maximum attention ﬂow from any
node in any of the layers to any of the input nodes.
We can use this maximum-ﬂow-value as an approximation of the attention to input nodes. In attention
ﬂow, the weight of a single path is the minimum
value of the weights of the edges in the path, instead of the product of the weights. Besides, we
attention rollout
 attention ﬂow
Figure 3: Attention maps for the CLS token
.
can not compute the attention for node sto node
tby adding up the weights of all paths between
these two nodes, since there might be an overlap
between the paths and this might result in overﬂow
in the overlapping edges.
It is noteworthy that both of the proposed methods can be computed in polynomial time. O(dn2)
for attention rollout and O(d2n4)for attention
ﬂow, where dis the depth of the model, and nis


for attention rollout and O(d2n4)for attention
ﬂow, where dis the depth of the model, and nis
the number of tokens.
4 Analysis and Discussion
Now, we take a closer look at these three views of
attention. Figure 1 depicts raw attention, attention
rollout and attention ﬂow for a correctly classiﬁed
example across different layers. It is noteworthy
that the ﬁrst layer of attention rollout and attention
ﬂow are the same, and their only difference with
raw attention is the addition of residual connections. As we move to the higher layers, we see that
the residual connections fade away. Moreover, in
contrast to raw attention, the patterns of attention
rollout and attention ﬂow become more distinctive
in the higher layers.
Figures 2 and 3 show the weights from raw attention, attention rollout and attention ﬂow for the
CLS embedding over input tokens (x-axis) in all
6 layers (y-axis) for three examples. The ﬁrst example is the same as the one in Figure 1. The second example is “ the article on NNP large systems
<?>”. The model correctly classiﬁes this example and changing the subject of the missing verb


<?>”. The model correctly classiﬁes this example and changing the subject of the missing verb
from “article” to “article s” ﬂips the decision of the
model. The third example is “ here the NNS differ
in that the female <?>”, which is a miss-classiﬁed
example and again changing “NNS” (plural noun)
to “NNP” (singular proper noun) ﬂips the decision
of the model.
For all cases, the raw attention weights are almost uniform above layer three (discussed before).
raw attention
attention rollout
attention ﬂow
(a) “The author talked to Sara about mask book .”
raw attention
attention rollout
attention ﬂow
(b)“Mary convinced John of mask love.”
Figure 4: Bert attention maps. We look at the attention
weights from the mask embedding to the two potential
references for it, e.g. “author” and “Sara” in (a) and
“Mary” and “John” in (b). The bars, at the left, show
the relative predicted probability for the two possible


“Mary” and “John” in (b). The bars, at the left, show
the relative predicted probability for the two possible
pronouns, “his” and “her”.
In the case of the correctly classiﬁed example, we
observe that both attention rollout and attention
ﬂow assign relatively high weights to both the subject of the verb, “article’ and the attractor, “systems”. For the miss-classiﬁed example, both attention rollout and attention ﬂow assign relatively
high scores to the “NNS” token which is not the
subject of the verb. This can explain the wrong
prediction of the model.
The main difference between attention rollout
and attention ﬂow is that attention ﬂow weights are
amortized among the set of most attended tokens,
as expected. Attention ﬂow can indicate a set of
input tokens that are important for the ﬁnal decision.
Thus we do not get sharp distinctions among them.
On the other hand, attention rollout weights are
more focused compared to attention ﬂow weights,
which is sensible for the third example but not as
much for the second one.
L1 L3 L5 L6


more focused compared to attention ﬂow weights,
which is sensible for the third example but not as
much for the second one.
L1 L3 L5 L6
Raw 0.120.21 0.090.21 0.080.20 0.090.21
Rollout 0.110.19 0.120.21 0.130.21 0.130.20
Flow 0.110.19 0.110.21 0.120.22 0.140.21
Table 3: SpearmanR correlation of attention based importance with input gradients for 100 samples from the
test set for the DistillBERT model ﬁne tuned on SST-2.
Furthermore, as shown in Table 1 and 2 both
attention rollout and attention ﬂow, are better
correlated with blank-out scores and input gradients compared to raw attention, but attention ﬂow
weights are more reliable than attention rollout.
The difference between these two methods is rooted
in their different views of attention weights. Attention ﬂow views them as capacities, and at every
step of the algorithm, it uses as much of the capacity as possible. Hence, attention ﬂow computes the
maximum possibility of token identities to propagate to the higher layers. Whereas attention rollout


maximum possibility of token identities to propagate to the higher layers. Whereas attention rollout
views them as proportion factors and at every step,
it allows token identities to be propagated to higher
layers exactly based on this proportion factors. This
makes attention rollout stricter than attention ﬂow,
and so we see that attention rollout provides us with
more focused attention patterns. However, since
we are making many simplifying assumptions, the
strictness of attention rollout does not lead to more
accurate results, and the relaxation of attention ﬂow
seems to be a useful property.
At last, to illustrate the application of attention ﬂow and attention rollout on different tasks
and different models, we examine them on two
pretrained BERT models. We use the models
available at https://github.com/huggingface/
transformers .
Table 3 shows the correlation of the importance
score obtained from raw attention, attention rollout
and attention ﬂow from a DistillBERT (Sanh et al.,
2019) model ﬁne-tuned to solve “SST-2” (Socher
et al., 2013), the sentiment analysis task from the
glue benchmark (Wang et al., 2018). Even though
for this model, all three methods have very low
correlation with the input gradients, we can still see


glue benchmark (Wang et al., 2018). Even though
for this model, all three methods have very low
correlation with the input gradients, we can still see
that attention rollout and attention ﬂow are slightly
better than raw attention.
Furthermore, in Figure 4, we show an example
of applying these methods to a pre-trained Bert
to see how it resolves the pronouns in a sentence.
What we do here is to feed the model with a sentence, masking a pronoun. Next, we look at the
prediction of the model for the masked word and
compare the probabilities assigned to “her” and
“his”. Then we look at raw attention, attention rollout and attention ﬂow weights of the embeddings
for the masked pronoun at all the layers. In the
ﬁrst example, in Figure 4a, attention rollout and
attention ﬂow are consistent with each other and
the prediction of the model. Whereas, the ﬁnal
layer of raw attention does not seem to be consistent with the prediction of the models, and it varies
a lot across different layers. In the second example, in Figure 4b, only attention ﬂow weights areconsistent with the prediction of the model.
5 Conclusion
Translating embedding attentions to token attentions can provide us with better explanations about


5 Conclusion
Translating embedding attentions to token attentions can provide us with better explanations about
models’ internals. Yet, we should be cautious about
our interpretation of these weights, because, we
are making many simplifying assumptions when
we approximate information ﬂow in a model with
the attention weights. Our ideas are simple and
task/architecture agnostic. In this paper, we insisted on sticking with simple ideas that only require attention weights and can be easily employed
in any task or architecture that uses self-attention.
We should note that all our analysis in this paper is
for a Transformer encoder, with no casual masking.
Since in Transformer decoder, future tokens are
masked, naturally there is more attention toward
initial tokens in the input sequence, and both attention rollout and attention ﬂow will be biased toward
these tokens. Hence, to apply these methods on a
Transformer decoder, we should ﬁrst normalize
based on the receptive ﬁeld of attention.
Following this work, we can build the attention
graph with effective attention weights (Brunner
et al., 2020) instead of raw attentions. Furthermore,
we can come up with a new method that adjusts the
attention weights using gradient-based attribution
methods (Ancona et al., 2019).
Acknowledgements


we can come up with a new method that adjusts the
attention weights using gradient-based attribution
methods (Ancona et al., 2019).
Acknowledgements
We thank Mostafa Dehghani, Wilker Aziz, and the
anonymous reviewers for their valuable feedback
and comments on this work. The work presented
here was funded by the Netherlands Organization
for Scientiﬁc Research (NWO), through a Gravitation Grant 024.001.006 to the Language in Interaction Consortium.
References
Marco Ancona, Enea Ceolini, Cengiz ¨Oztireli, and
Markus Gross. 2019. Gradient-Based Attribution
Methods , pages 169–191. Springer International
Publishing.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In proceedings of
the 2015 International Conference on Learning Representations .
Gino Brunner, Yang Liu, Damian Pascual, Oliver
Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On identiﬁability in transformers. In
International Conference on Learning Representations .
Hanjie Chen and Yangfeng Ji. 2019. Improving the interpretability of neural sentiment classiﬁers via data


International Conference on Learning Representations .
Hanjie Chen and Yangfeng Ji. 2019. Improving the interpretability of neural sentiment classiﬁers via data
augmentation. arXiv preprint arXiv:1909.04225 .
Kevin Clark, Urvashi Khandelwal, Omer Levy, and
Christopher D. Manning. 2019. What does BERT
look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP:
Analyzing and Interpreting Neural Networks for
NLP, pages 276–286, Florence, Italy. Association
for Computational Linguistics.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim,
Adam Pearce, Fernanda Vi ´egas, and Martin Wattenberg. 2019. Visualizing and measuring the geometry
of bert. arXiv preprint arXiv:1906.02715 .
Thomas H. Cormen, Charles E. Leiserson, Ronald L.
Rivest, and Clifford Stein. 2009. Introduction to
Algorithms, Third Edition , 3rd edition. The MIT
Press.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals,


Algorithms, Third Edition , 3rd edition. The MIT
Press.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals,
Jakob Uszkoreit, and Łukasz Kaiser. 2019. Universal transformers. In proceedings of the 2019 International Conference on Learning Representations .
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of
deep bidirectional transformers for language understanding. In proceedings of the 2019 Conference
of the North American Chapter of the Association
for Computational Linguistics: Human Language
Technologies . Association for Computational Linguistics.
Sarthak Jain and Byron C. Wallace. 2019. Attention
is not Explanation. In proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3543–3556. Association
for Computational Linguistics.
Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim.
2017. Interactive visualization and manipulation
of attention-based neural machine translation. In
proceedings of the 2017 Conference on Empirical
Methods in Natural Language Processing: System
Demonstrations , pages 121–126.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.