deep neural networks. We evaluate its ability to reason about several challenging
physical domains: n-body problems, rigid-body collision, and non-rigid dynamics.
Our results show it can be trained to accurately simulate the physical trajectories of
dozens of objects over thousands of time steps, estimate abstract quantities such
as energy, and generalize automatically to systems with different numbers and
conﬁgurations of objects and relations. Our interaction network implementation
is the ﬁrst general-purpose, learnable physics engine, and a powerful general
framework for reasoning about object and relations in a wide variety of complex
real-world domains.
1 Introduction
Representing and reasoning about objects, relations and physics is a “core” domain of human common
sense knowledge [ 26], and among the most basic and important aspects of intelligence [ 28,15]. Many
everyday problems, such as predicting what will happen next in physical environments or inferring
underlying properties of complex scenes, are challenging because their elements can be composed
in combinatorially many possible arrangements. People can nevertheless solve such problems by
decomposing the scenario into distinct objects and relations, and reasoning about the consequences
of their interactions and dynamics. Here we introduce the interaction network – a model that can
perform an analogous form of reasoning about objects and relations in complex systems.
Interaction networks combine three powerful approaches: structured models, simulation, and deep

Published in NIPS 2016


perform an analogous form of reasoning about objects and relations in complex systems.
Interaction networks combine three powerful approaches: structured models, simulation, and deep
learning. Structured models [ 7] can exploit rich, explicit knowledge of relations among objects,
independent of the objects themselves, which supports general-purpose reasoning across diverse
contexts. Simulation is an effective method for approximating dynamical systems, predicting how the
elements in a complex system are inﬂuenced by interactions with one another, and by the dynamics
of the system. Deep learning [ 24,16] couples generic architectures with efﬁcient optimization
algorithms to provide highly scalable learning and inference in challenging real-world settings.arXiv:1612.00222v1  [cs.AI]  1 Dec 2016
Object reasoning Relational reasoning
Compute interaction Apply object dynamicsEffectsObjects,
relationsPredictions,
inferences
a.
b.Figure 1: Schematic of an interaction network. a.For physical reasoning, the model takes objects and relations
as input, reasons about their interactions, and applies the effects and physical dynamics to predict new states. b.
For more complex systems, the model takes as input a graph that represents a system of objects, oj, and relations,

Interaction Networks for Learning about Objects, Relations and Physics

Summary:
Reasoning about objects, relations, and physics is central to human
intelligence, and a key goal of artificial intelligence. Here we introduce the
interaction network, a model which can reason about how objects in complex
systems interact, supporting dynamical predictions, as well as inferences about
the abstract properties of the system. Our model takes graphs as input,
performs object- and relation-centric reasoning in a way that is analogous to a
simulation, and is implemented using deep neural networks. We evaluate its
ability to reason about several challenging physical domains: n-body problems,
rigid-body collision, and non-rigid dynamics. Our results show it can be
trained to accurately simulate the physical trajectories of dozens of objects
over thousands of time steps, estimate abstract quantities such as energy, and
generalize automatically to systems with different numbers and configurations
of objects and relations. Our interaction network implementation is the first
general-purpose, learnable physics engine, and a powerful general framework for
reasoning about object and relations in a wide variety of complex real-world
domains.

Peter W. Battaglia
Razvan Pascanu
Matthew Lai
Danilo Rezende
Koray Kavukcuoglu

Published in NIPS 2016


For more complex systems, the model takes as input a graph that represents a system of objects, oj, and relations,
hi; j; r kik, instantiates the pairwise interaction terms, bk, and computes their effects, ek, via a relational model,
fR(). The ekare then aggregated and combined with the ojand external effects, xj, to generate input (as cj),
for an object model, fO(), which predicts how the interactions and dynamics inﬂuence the objects, p.
Interaction networks explicitly separate how they reason about relations from how they reason about
objects, assigning each task to distinct models which are: fundamentally object- and relation-centric;
and independent of the observation modality and task speciﬁcation (see Model section 2 below
and Fig. 1a). This lets interaction networks automatically generalize their learning across variable
numbers of arbitrarily ordered objects and relations, and also recompose their knowledge of entities
and interactions in novel and combinatorially many ways. They take relations as explicit input,
allowing them to selectively process different potential interactions for different input data, rather
than being forced to consider every possible interaction or those imposed by a ﬁxed architecture.

Published in NIPS 2016


allowing them to selectively process different potential interactions for different input data, rather
than being forced to consider every possible interaction or those imposed by a ﬁxed architecture.
We evaluate interaction networks by testing their ability to make predictions and inferences about various physical systems, including n-body problems, and rigid-body collision, and non-rigid dynamics.
Our interaction networks learn to capture the complex interactions that can be used to predict future
states and abstract physical properties, such as energy. We show that they can roll out thousands of
realistic future state predictions, even when trained only on single-step predictions. We also explore
how they generalize to novel systems with different numbers and conﬁgurations of elements. Though
they are not restricted to physical reasoning, the interaction networks used here represent the ﬁrst
general-purpose learnable physics engine, and even have the potential to learn novel physical systems
for which no physics engines currently exist.
Related work Our model draws inspiration from previous work that reasons about graphs and
relations using neural networks. The “graph neural network” [ 23,19] is a framework that shares
learning across nodes and edges, the “recursive autoencoder” [ 25] adapts its processing architecture
to exploit an input parse tree, the “neural programmer-interpreter” [ 22] is a composable network that

Published in NIPS 2016


to exploit an input parse tree, the “neural programmer-interpreter” [ 22] is a composable network that
mimics the execution trace of a program, and the “spatial transformer” [ 11] learns to dynamically
modify network connectivity to capture certain types of interactions. Others have explored deep
learning of logical and arithmetic relations [ 27], and relations suitable for visual question-answering
[1].
The behavior of our model is similar in spirit to a physical simulation engine [ 2], which generates
sequences of states by repeatedly applying rules that approximate the effects of physical interactions
and dynamics on objects over time. The interaction rules are relation-centric, operating on two or
more objects that are interacting, and the dynamics rules are object-centric, operating on individual
objects and the aggregated effects of the interactions they participate in. A similar approach has been
developed independently by Chang et al. [4].
Previous AI work on physical reasoning explored commonsense knowledge, qualitative representations, and simulation techniques for approximating physical prediction and inference [ 29,9] . The
“NeuroAnimator” [ 8] was perhaps the ﬁrst quantitative approach to learning physical dynamics, by
training neural networks to predict and control the state of articulated bodies. Ladický et al. [ 14]

Published in NIPS 2016


training neural networks to predict and control the state of articulated bodies. Ladický et al. [ 14]
recently used regression forests to learn ﬂuid dynamics. Recent advances in convolutional neural
networks (CNNs) have led to efforts that learn to predict coarse-grained physical dynamics from
2
images [ 20,17,18]. Notably, Fragkiadaki et al. [ 6] used CNNs to predict and control a moving
ball from an image centered at its coordinates. Mottaghi et al. [ 21] trained CNNs to predict the 3D
trajectory of an object after an external impulse is applied. Wu et al. [ 30] used CNNs to parse objects
from images, which were then input to a physics engine that supported prediction and inference.
2 Model
Deﬁnition To describe our model, we use physical reasoning as an example (Fig. 1a), and build
from a simple model to the full interaction network (abbreviated IN). To predict the dynamics of a
single object, one might use an object-centric function, fO, which inputs the object’s state, ot, at
timet, and outputs a future state, ot+1. If two or more objects are governed by the same dynamics,
fOcould be applied to each, independently, to predict their respective future states. But if the

Published in NIPS 2016


fOcould be applied to each, independently, to predict their respective future states. But if the
objects interact with one another, then fOis insufﬁcient because it does not capture their relationship.
Assuming two objects and one directed relationship, e.g., a ﬁxed object attached by a spring to a freely
moving mass, the ﬁrst (the sender ,o1) inﬂuences the second (the receiver ,o2) via their interaction.
The effect of this interaction, et+1, can be predicted by a relation-centric function, fR. ThefRtakes
as inputo1,o2, as well as attributes of their relationship, r, e.g., the spring constant. The fOis
modiﬁed so it can input both et+1and the receiver’s current state, o2;t, enabling the interaction to
inﬂuence its future state, o2;t+1,
et+1=fR(o1;t;o2;t;r) o2;t+1=fO(o2;t;et+1)
The above formulation can be expanded to larger and more complex systems by representing them

Published in NIPS 2016


The above formulation can be expanded to larger and more complex systems by representing them
as a graph,G=hO;Ri, where the nodes, O, correspond to the objects, and the edges, R, to the
relations (see Fig. 1b). We assume an attributed, directed multigraph because the relations have
attributes, and there can be multiple distinct relations between two objects (e.g., rigid and magnetic
interactions). For a system with NOobjects andNRrelations, the inputs to the IN are,
O=fojgj=1:::NO; R=fhi;j;r kikgk=1:::NRwherei6=j;1i;jNO; X=fxjgj=1:::NO
TheOrepresents the states of each object. The triplet, hi;j;r kik, represents the k-th relation in the
system, from sender, oi, to receiver, oj, with relation attribute, rk. TheXrepresents external effects,
such as active control inputs or gravitational acceleration, which we deﬁne as not being part of the
system, and which are applied to each object separately.
The basic IN is deﬁned as,

Published in NIPS 2016


system, and which are applied to each object separately.
The basic IN is deﬁned as,
IN(G) =O(a(G; X;  R(m(G) ) )) (1)
m(G) =B=fbkgk=1:::NR
fR(bk) =ek
R(B) =E=fekgk=1:::NRa(G;X;E ) =C=fcjgj=1:::NO
fO(cj) =pj
O(C) =P=fpjgj=1:::NO(2)
The marshalling function, m, rearranges the objects and relations into interaction terms, bk=
hoi;oj;rki2B, one per relation, which correspond to each interaction’s receiver, sender, and
relation attributes. The relational model, R, predicts the effect of each interaction, ek2E, by
applyingfRto eachbk. The aggregation function, a, collects all effects, ek2E, that apply to each
receiver object, merges them, and combines them with OandXto form a set of object model inputs,

Published in NIPS 2016


receiver object, merges them, and combines them with OandXto form a set of object model inputs,
cj2C, one per object. The object model, O, predicts how the interactions and dynamics inﬂuence
the objects by applying fOto eachcj, and returning the results, pj2P. This basic IN can predict
the evolution of states in a dynamical system – for physical simulation, Pmay equal the future states
of the objects, Ot+1.
The IN can also be augmented with an additional component to make abstract inferences about the
system. The pj2P, rather than serving as output, can be combined by another aggregation function,
g, and input to an abstraction model, A, which returns a single output, q, for the whole system. We
explore this variant in our ﬁnal experiments that use the IN to predict potential energy.
An IN applies the same fRandfOto everybkandcj, respectively, which makes their relational and
object reasoning able to handle variable numbers of arbitrarily ordered objects and relations. But
one additional constraint must be satisﬁed to maintain this: the afunction must be commutative and
3
associative over the objects and relations. Using summation within ato merge the elements of Einto

Published in NIPS 2016


3
associative over the objects and relations. Using summation within ato merge the elements of Einto
Csatisﬁes this, but division would not.
Here we focus on binary relations, which means there is one interaction term per relation, but another
option is to have the interactions correspond to n-th order relations by combining nsenders in each bk.
The interactions could even have variable order, where each bkincludes all sender objects that interact
with a receiver, but would require a fRthan can handle variable-length inputs. These possibilities are
beyond the scope of this work, but are interesting future directions.
Implementation The general deﬁnition of the IN in the previous section is agnostic to the choice
of functions and algorithms, but we now outline a learnable implementation capable of reasoning
about complex systems with nonlinear relations and dynamics. We use standard deep neural network
building blocks, multilayer perceptrons (MLP), matrix operations, etc., which can be trained efﬁciently
from data using gradient-based optimization, such as stochastic gradient descent.
We deﬁneOas aDSNOmatrix, whose columns correspond to the objects’ DS-length state vectors.

Published in NIPS 2016


We deﬁneOas aDSNOmatrix, whose columns correspond to the objects’ DS-length state vectors.
The relations are a triplet, R=hRr;Rs;Rai, whereRrandRsareNONRbinary matrices which
index the receiver and sender objects, respectively, and Rais aDRNRmatrix whose DR-length
columns represent the NRrelations’ attributes. The j-th column of Rris a one-hot vector which
indicates the receiver object’s index; Rsindicates the sender similarly. For the graph in Fig. 1b,
Rr=h0 0
1 1
0 0i
andRs=h1 0
0 0
0 1i
. TheXis aDXNOmatrix, whose columns are DX-length vectors
that represent the external effect applied each of the NOobjects.
The marshalling function, m, computes the matrix products, ORrandORs, and concatenates them
withRa:m(G) = [ORr;ORs;Ra] =B.
The resulting Bis a(2DS+DR)NRmatrix, whose columns represent the interaction terms, bk,
for theNRrelations (we denote vertical and horizontal matrix concatenation with a semicolon and

Published in NIPS 2016


for theNRrelations (we denote vertical and horizontal matrix concatenation with a semicolon and
comma, respectively). The way mconstructs interaction terms can be modiﬁed, as described in our
Experiments section (3).
TheBis input toR, which applies fR, an MLP, to each column. The output of fRis aDE-length
vector,ek, a distributed representation of the effects. The Rconcatenates the NReffects to form the
DENReffect matrix, E.
TheG,X, andEare input toa, which computes the DENOmatrix product, E=ERT
r, whose
j-th column is equivalent to the elementwise sum across all ekwhose corresponding relation has
receiver object, j. The Eis concatenated with OandX:a(G;X;E ) = [O;X;E] =C.
The resulting Cis a(DS+DX+DE)NOmatrix, whose NOcolumns represent the object states,
external effects, and per-object aggregate interaction effects.
TheCis input toO, which applies fO, another MLP, to each of the NOcolumns. The output of fO

Published in NIPS 2016


TheCis input toO, which applies fO, another MLP, to each of the NOcolumns. The output of fO
is aDP-length vector, pj, andOconcatenates them to form the output matrix, P.
To infer abstract properties of a system, an additional Ais appended and takes Pas input. The g
aggregation function performs an elementwise sum across the columns of Pto return aDP-length
vector, P. The Pis input toA, another MLP, which returns a DA-length vector, q, that represents
an abstract, global property of the system.
Training an IN requires optimizing an objective function over the learnable parameters of RandO.
Note,mandainvolve matrix operations that do not contain learnable parameters.
BecauseRandOare shared across all relations and objects, respectively, training them is statistically efﬁcient. This is similar to CNNs, which are very efﬁcient due to their weight-sharing scheme.
A CNN treats a local neighborhood of pixels as related, interacting entities: each pixel is effectively
a receiver object and its neighboring pixels are senders. The convolution operator is analogous to
R, wherefRis the local linear/nonlinear kernel applied to each neighborhood. Skip connections,

Published in NIPS 2016


R, wherefRis the local linear/nonlinear kernel applied to each neighborhood. Skip connections,
recently popularized by residual networks, are loosely analogous to how the IN inputs Oto both
RandO, though in CNNs relation- and object-centric reasoning are not delineated. But because
CNNs exploit local interactions in a ﬁxed way which is well-suited to the speciﬁc topology of images,
capturing longer-range dependencies requires either broad, insensitive convolution kernels, or deep
stacks of layers, in order to implement sufﬁciently large receptive ﬁelds. The IN avoids this restriction
by being able to process arbitrary neighborhoods that are explicitly speciﬁed by the Rinput.
4
3 Experiments
Physical reasoning tasks Our experiments explored two types of physical reasoning tasks: predicting future states of a system, and estimating their abstract properties, speciﬁcally potential energy.
We evaluated the IN’s ability to learn to make these judgments in three complex physical domains:
n-body systems; balls bouncing in a box; and strings composed of springs that collide with rigid
objects. We simulated the 2D trajectories of the elements of these systems with a physics engine, and
recorded their sequences of states. See the Supplementary Material for full details.

Published in NIPS 2016


objects. We simulated the 2D trajectories of the elements of these systems with a physics engine, and
recorded their sequences of states. See the Supplementary Material for full details.
In the n-body domain, such as solar systems, all nbodies exert distance- and mass-dependent
gravitational forces on each other, so there were n(n1)relations input to our model. Across
simulations, the objects’ masses varied, while all other ﬁxed attributes were held constant. The
training scenes always included 6 bodies, and for testing we used 3, 6, and 12 bodies. In half of
the systems, bodies were initialized with velocities that would cause stable orbits, if not for the
interactions with other objects; the other half had random velocities.
In the bouncing balls domain, moving balls could collide with each other and with static walls.
The walls were represented as objects whose shape attribute represented a rectangle, and whose
inverse-mass was 0. The relations input to the model were between the nobjects (which included the
walls), for (n(n1)relations). Collisions are more difﬁcult to simulate than gravitational forces, and
the data distribution was much more challenging: each ball participated in a collision on less than 1%

Published in NIPS 2016


the data distribution was much more challenging: each ball participated in a collision on less than 1%
of the steps, following straight-line motion at all other times. The model thus had to learn that despite
there being a rigid relation between two objects, they only had meaningful collision interactions when
they were in contact. We also varied more of the object attributes – shape, scale and mass (as before)
– as well as the coefﬁcient of restitution, which was a relation attribute. Training scenes contained 6
balls inside a box with 4 variably sized walls, and test scenes contained either 3, 6, or 9 balls.
The string domain used two types of relations (indicated in rk), relation structures that were more
sparse and speciﬁc than all-to-all, as well as variable external effects. Each scene contained a string,
comprised of masses connected by springs, and a static, rigid circle positioned below the string. The
nmasses had spring relations with their immediate neighbors ( 2(n1)), and all masses had rigid
relations with the rigid object ( 2n). Gravitational acceleration, with a magnitude that was varied
across simulation runs, was applied so that the string always fell, usually colliding with the static
object. The gravitational acceleration was an external input (not to be confused with the gravitational

Published in NIPS 2016


across simulation runs, was applied so that the string always fell, usually colliding with the static
object. The gravitational acceleration was an external input (not to be confused with the gravitational
attraction relations in the n-body experiments). Each training scene contained a string with 15 point
masses, and test scenes contained either 5, 15, or 30 mass strings. In training, one of the point masses
at the end of the string, chosen at random, was always held static, as if pinned to the wall, while the
other masses were free to move. In the test conditions, we also included strings that had both ends
pinned, and no ends pinned, to evaluate generalization.
Our model takes as input the state of each system, G, decomposed into the objects, O(e.g., n-body
objects, balls, walls, points masses that represented string elements), and their physical relations, R
(e.g., gravitational attraction, collisions, springs), as well as the external effects, X(e.g., gravitational
acceleration). Each object state, oj, could be further divided into a dynamic state component
(e.g., position and velocity) and a static attribute component (e.g., mass, size, shape). The relation
attributes,Ra, represented quantities such as the coefﬁcient of restitution, and spring constant. The

Published in NIPS 2016


attributes,Ra, represented quantities such as the coefﬁcient of restitution, and spring constant. The
input represented the system at the current time. The prediction experiment’s target outputs were the
velocities of the objects on the subsequent time step, and the energy estimation experiment’s targets
were the potential energies of the system on the current time step. We also generated multi-step
rollouts for the prediction experiments (Fig. 2), to assess the model’s effectiveness at creating visually
realistic simulations. The output velocity, vt, on time step tbecame the input velocity on t+ 1, and
the position at t+ 1was updated by the predicted velocity at t.
Data Each of the training, validation, test data sets were generated by simulating 2000 scenes
over 1000 time steps, and randomly sampling 1 million, 200k, and 200k one-step input/target pairs,
respectively. The model was trained for 2000 epochs, randomly shufﬂing the data indices between
each. We used mini-batches of 100, and balanced their data distributions so the targets had similar
per-element statistics. The performance reported in the Results was measured on held-out test data.
We explored adding a small amount of Gaussian noise to 20% of the data’s input positions and

Published in NIPS 2016


We explored adding a small amount of Gaussian noise to 20% of the data’s input positions and
velocities during the initial phase of training, which was reduced to 0% from epochs 50 to 250. The
5
noise std. dev. was 0:05the std. dev. of each element’s values across the dataset. It allowed the
model to experience physically impossible states which could not have been generated by the physics
engine, and learn to project them back to nearby, possible states. Our error measure did not reﬂect
clear differences with or without noise, but rollouts from models trained with noise were slightly
more visually realistic, and static objects were less subject to drift over many steps.
Model architecture ThefRandfOMLPs contained multiple hidden layers of linear transforms
plus biases, followed by rectiﬁed linear units (ReLUs), and an output layer that was a linear transform
plus bias. The best model architecture was selected by a grid search over layer sizes and depths. All
inputs (except RrandRs) were normalized by centering at the median and rescaling the 5th and 95th
percentiles to -1 and 1. All training objectives and test measures used mean squared error (MSE)
between the model’s prediction and the ground truth target.

Published in NIPS 2016


percentiles to -1 and 1. All training objectives and test measures used mean squared error (MSE)
between the model’s prediction and the ground truth target.
All prediction experiments used the same architecture, with parameters selected by a hyperparameter
search. The fRMLP had four, 150-length hidden layers, and output length DE= 50 . ThefOMLP
had one, 100-length hidden layer, and output length DP= 2, which targeted the x;y-velocity. The
mandawere customized so that the model was invariant to the absolute positions of objects in the
scene. Themconcatenated three terms for each bk: the difference vector between the dynamic states
of the receiver and sender, the concatenated receiver and sender attribute vectors, and the relation
attribute vector. The aonly outputs the velocities, not the positions, for input to O.
The energy estimation experiments used the IN from the prediction experiments with an additional
AMLP which had one, 25-length hidden layer. Its Pinputs’ columns were length DP= 10 , and
its output length was DA= 1.
We optimized the parameters using Adam [ 13], with a waterfall schedule that began with a learning
rate of 0:001and down-scaled the learning rate by 0:8each time the validation error, estimated over

Published in NIPS 2016


rate of 0:001and down-scaled the learning rate by 0:8each time the validation error, estimated over
a window of 40epochs, stopped decreasing.
Two forms of L2 regularization were explored: one applied to the effects, E, and another to the model
parameters. Regularizing Eimproved generalization to different numbers of objects and reduced
drift over many rollout steps. It likely incentivizes sparser communication between the RandO,
prompting them to operate more independently. Regularizing the parameters generally improved
performance and reduced overﬁtting. Both penalty factors were selected by a grid search.
Few competing models are available in the literature to compare our model against, but we considered
several alternatives: a constant velocity baseline which output the input velocity; an MLP baseline,
with two 300-length hidden layers, which took as input a ﬂattened vector of all of the input data; and
a variant of the IN with the Rcomponent removed (the interaction effects, E, was set to a 0-matrix).
4 Results
Prediction experiments Our results show that the IN can predict the next-step dynamics of our task
domains very accurately after training, with orders of magnitude lower test error than the alternative

Published in NIPS 2016


4 Results
Prediction experiments Our results show that the IN can predict the next-step dynamics of our task
domains very accurately after training, with orders of magnitude lower test error than the alternative
models (Fig. 3a, d and g, and Table 2). Because the dynamics of each domain depended crucially on
interactions among objects, the IN was able to learn to exploit these relationships for its predictions.
The dynamics-only IN had no mechanism for processing interactions, and performed similarly to the
constant velocity model. The baseline MLP’s connectivity makes it possible, in principle, for it to
learn the interactions, but that would require learning how to use the relation indices to selectively
process the interactions. It would also not beneﬁt from sharing its learning across relations and
objects, instead being forced to approximate the interactive dynamics in parallel for each objects.
The IN also generalized well to systems with fewer and greater numbers of objects (Figs. 3b-c, e-f
and h-k, and Table SM1 in Supp. Mat.). For each domain, we selected the best IN model from the
system size on which it was trained, and evaluated its MSE on a different system size. When tested
on smaller n-body and spring systems from those on which it was trained, its performance actually

Published in NIPS 2016


on smaller n-body and spring systems from those on which it was trained, its performance actually
exceeded a model trained on the smaller system. This may be due to the model’s ability to exploit its
greater experience with how objects and relations behave, available in the more complex system.
We also found that the IN trained on single-step predictions can be used to simulate trajectories over
thousands of steps very effectively, often tracking the ground truth closely, especially in the n-body
and string domains. When rendered into images and videos, the model-generated trajectories are
6
True Model True Model True ModelTime Time TimeFigure 2: Prediction rollouts. Each column contains three panels of three video frames (with motion blur),
each spanning 1000 rollout steps. Columns 1-2 are ground truth and model predictions for n-body systems, 3-4
are bouncing balls, and 5-6 are strings. Each model column was generated by a single model, trained on the
underlying states of a system of the size in the top panel. The middle and bottom panels show its generalization
to systems of different sizes and structure. For n-body, the training was on 6 bodies, and generalization was to 3
and 12 bodies. For balls, the training was on 6 balls, and generalization was to 3 and 9 balls. For strings, the

Published in NIPS 2016


and 12 bodies. For balls, the training was on 6 balls, and generalization was to 3 and 9 balls. For strings, the
training was on 15 masses with 1 end pinned, and generalization was to 30 masses with 0 and 2 ends pinned.
The URLs to the full videos of each rollout are in Table 1.
7
Table 1: Rollout video URLs
System True Model
n-body 6 https://youtu.be/otIGNTFJwpU https://youtu.be/CXzubiwu4GM
n-body 3 https://youtu.be/zOTVzziJz24 https://youtu.be/3bFrbjdTc6s
n-body 12 https://youtu.be/JgSRaYmQyNY https://youtu.be/Rl44jdaIIiw
Balls 6 https://youtu.be/9c-UPXSjOHI https://youtu.be/SV741g_qp8M
Balls 3 https://youtu.be/op2d9IHh3Ak https://youtu.be/i0lOn_v8xVg
Balls 9 https://youtu.be/oQoquoVJ8V0 https://youtu.be/41y_saKvi3s

Published in NIPS 2016


Balls 9 https://youtu.be/oQoquoVJ8V0 https://youtu.be/41y_saKvi3s
String 15, 1 pinned https://youtu.be/xiz49xOPX9c https://youtu.be/pbk6VM_XQb4
String 30, 0 pinned https://youtu.be/TpixFr_ekAE https://youtu.be/tsE7cev0w08
String 30, 2 pinned https://youtu.be/x6g1ogRG31o https://youtu.be/ZDCZyqyrv5o
10-2
10-3g. 15, 1h. 5, 1i. 30, 1j. 15, 0k. 15, 2String
1
10-110102MSE (log-scale)a. 6b. 3c. 12n-body
10-210-1
10-3d. 6e. 3f. 9Balls
IN (15 obj, 1 pin) IN (5 obj, 1 pin)
IN (15 obj, 0 pin)IN (30 obj, 1 pin)
IN (15 obj, 2 pin)IN (3 obj) IN (12 obj) IN (6 obj)

Published in NIPS 2016


IN (15 obj, 0 pin)IN (30 obj, 1 pin)
IN (15 obj, 2 pin)IN (3 obj) IN (12 obj) IN (6 obj)
Constant velocity Baseline MLP Dynamics-only ININ (3 obj) IN (9 obj) IN (6 obj)10-2
Figure 3: Prediction experiment accuracy and generalization. Each colored bar represents the MSE between a
model’s predicted velocity and the ground truth physics engine’s (the y-axes are log-scaled). Sublots ( a-c) show
n-body performance, ( d-f) show balls, and ( g-k) show string. The leftmost subplots in each ( a, d, g ) for each
domain compare the constant velocity model (black), baseline MLP (grey), dynamics-only IN (red), and full IN
(blue). The other panels show the IN’s generalization performance to different numbers and conﬁgurations of
objects, as indicated by the subplot titles. For the string systems, the numbers correspond to: (the number of
masses, how many ends were pinned).
usually visually indistinguishable from those of the ground truth physics engine (Fig. 2; see Table 1
for URLs to the full videos of each rollout). This is not to say that given the same initial conditions,

Published in NIPS 2016


for URLs to the full videos of each rollout). This is not to say that given the same initial conditions,
they cohere perfectly: the dynamics are highly nonlinear and imperceptible prediction errors by the
model can rapidly lead to large differences in the systems’ states. But the incoherent rollouts do
not violate people’s expectations, and might be roughly on par with people’s understanding of these
domains.
Estimating abstract properties We trained an abstract-estimation variant of our model to predict
potential energies in the n-body and string domains (the ball domain’s potential energies were always
0), and found it was much more accurate (n-body MSE 1:4, string MSE 1:1) than the MLP baseline
(n-body MSE 19, string MSE 425). The IN presumably learns the gravitational and spring potential
energy functions, applies them to the relations in their respective domains, and combines the results.
5 Discussion
We introduced interaction networks as a ﬂexible and efﬁcient model for explicit reasoning about
objects and relations in complex systems. Our results provide surprisingly strong evidence of their
ability to learn accurate physical simulations and generalize their training to novel systems with
different numbers and conﬁgurations of objects and relations. They could also learn to infer abstract

Published in NIPS 2016


ability to learn accurate physical simulations and generalize their training to novel systems with
different numbers and conﬁgurations of objects and relations. They could also learn to infer abstract
properties of physical systems, such as potential energy. The alternative models we tested performed
much more poorly, with orders of magnitude greater error. Simulation over rich mental models is
thought to be a crucial mechanism of how humans reason about physics and other complex domains
[5,12,10], and Battaglia et al. [ 3] recently posited a simulation-based “intuitive physics engine”
model to explain human physical scene understanding. Our interaction network implementation is the
ﬁrst learnable physics engine that can scale up to real-world problems, and is a promising template for
new AI approaches to reasoning about other physical and mechanical systems, scene understanding,
social perception, hierarchical planning, and analogical reasoning.
In the future, it will be important to develop techniques that allow interaction networks to handle
very large systems with many interactions, such as by culling interaction computations that will have
8
Table 2: Prediction experiment MSEs
Domain Constant velocity Baseline Dynamics-only IN IN
n-body 82 79 76 0.25
Balls 0.074 0.072 0.074 0.0020
String 0.018 0.016 0.017 0.0011

Published in NIPS 2016


Balls 0.074 0.072 0.074 0.0020
String 0.018 0.016 0.017 0.0011
negligible effects. The interaction network may also serve as a powerful model for model-predictive
control inputting active control signals as external effects – because it is differentiable, it naturally
supports gradient-based planning. It will also be important to prepend a perceptual front-end that
can infer from objects and relations raw observations, which can then be provided as input to an
interaction network that can reason about the underlying structure of a scene. By adapting the
interaction network into a recurrent neural network, even more accurate long-term predictions might
be possible, though preliminary tests found little beneﬁt beyond its already-strong performance.
By modifying the interaction network to be a probabilistic generative model, it may also support
probabilistic inference over unknown object properties and relations.
By combining three powerful tools from the modern machine learning toolkit – relational reasoning
over structured knowledge, simulation, and deep learning – interaction networks offer ﬂexible,
accurate, and efﬁcient learning and inference in challenging domains. Decomposing complex
systems into objects and relations, and reasoning about them explicitly, provides for combinatorial

Published in NIPS 2016


accurate, and efﬁcient learning and inference in challenging domains. Decomposing complex
systems into objects and relations, and reasoning about them explicitly, provides for combinatorial
generalization to novel contexts, one of the most important future challenges for AI, and a crucial
step toward closing the gap between how humans and machines think.
References
[1]J Andreas, M Rohrbach, T Darrell, and D Klein. Learning to compose neural networks for question
answering. NAACL , 2016.
[2]D Baraff. Physically based modeling: Rigid body simulation. SIGGRAPH Course Notes, ACM SIGGRAPH ,
2(1), 2001.
[3]PW Battaglia, JB Hamrick, and JB Tenenbaum. Simulation as an engine of physical scene understanding.
Proceedings of the National Academy of Sciences , 110(45):18327–18332, 2013.
[4]Michael B. Chang, Tomer Ullman, Antonio Torralba, and Joshua B. Tenenbaum. A compositional
object-based approach to learning physical dynamics. Under Review , 2016.
[5] K.J.W. Craik. The nature of explanation . Cambridge University Press, 1943.

Published in NIPS 2016


object-based approach to learning physical dynamics. Under Review , 2016.
[5] K.J.W. Craik. The nature of explanation . Cambridge University Press, 1943.
[6]K Fragkiadaki, P Agrawal, S Levine, and J Malik. Learning visual predictive models of physics for playing
billiards. ICLR , 2016.
[7]Z. Ghahramani. Probabilistic machine learning and artiﬁcial intelligence. Nature , 521(7553):452–459,
2015.
[8]R Grzeszczuk, D Terzopoulos, and G Hinton. Neuroanimator: Fast neural network emulation and control of
physics-based models. In Proceedings of the 25th annual conference on Computer graphics and interactive
techniques , pages 9–20. ACM, 1998.
[9]P.J Hayes. The naive physics manifesto . Université de Genève, Institut pour les études s é mantiques et
cognitives, 1978.
[10] M. Hegarty. Mechanical reasoning by mental simulation. TICS , 8(6):280–285, 2004.
[11] M Jaderberg, K Simonyan, and A Zisserman. Spatial transformer networks. In NIPS , pages 2008–2016,
2015.

Published in NIPS 2016


[11] M Jaderberg, K Simonyan, and A Zisserman. Spatial transformer networks. In NIPS , pages 2008–2016,
2015.
[12] P.N. Johnson-Laird. Mental models: towards a cognitive science of language, inference, and consciousness ,
volume 6. Cambridge University Press, 1983.
[13] D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR , 2015.
[14] L Ladický, S Jeong, B Solenthaler, M Pollefeys, and M Gross. Data-driven ﬂuid simulations using
regression forests. ACM Transactions on Graphics (TOG) , 34(6):199, 2015.
[15] B Lake, T Ullman, J Tenenbaum, and S Gershman. Building machines that learn and think like people.
arXiv:1604.00289 , 2016.
[16] Y LeCun, Y Bengio, and G Hinton. Deep learning. Nature , 521(7553):436–444, 2015.
9
[17] A Lerer, S Gross, and R Fergus. Learning physical intuition of block towers by example. arXiv:1603.01312 ,
2016.

Published in NIPS 2016


9
[17] A Lerer, S Gross, and R Fergus. Learning physical intuition of block towers by example. arXiv:1603.01312 ,
2016.
[18] W Li, S Azimi, A Leonardis, and M Fritz. To fall or not to fall: A visual approach to physical stability
prediction. arXiv:1604.00066 , 2016.
[19] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks.
arXiv preprint arXiv:1511.05493 , 2015.
[20] R Mottaghi, H Bagherinezhad, M Rastegari, and A Farhadi. Newtonian image understanding: Unfolding
the dynamics of objects in static images. arXiv:1511.04048 , 2015.
[21] R Mottaghi, M Rastegari, A Gupta, and A Farhadi. " what happens if..." learning to predict the effect of
forces in images. arXiv:1603.05600 , 2016.
[22] SE Reed and N de Freitas. Neural programmer-interpreters. ICLR , 2016.

Published in NIPS 2016


[22] SE Reed and N de Freitas. Neural programmer-interpreters. ICLR , 2016.
[23] F. Scarselli, M. Gori, A.C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model.
IEEE Trans. Neural Networks , 20(1):61–80, 2009.
[24] J. Schmidhuber. Deep learning in neural networks: An overview. Neural Networks , 61:85–117, 2015.
[25] R Socher, E Huang, J Pennin, C Manning, and A Ng. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In in NIPS , pages 801–809, 2011.
[26] E Spelke, K Breinlinger, J Macomber, and K Jacobson. Origins of knowledge. Psychol. Rev. , 99(4):605–
632, 1992.
[27] I Sutskever and GE Hinton. Using matrices to model symbolic relationship. In D. Koller, D. Schuurmans,
Y . Bengio, and L. Bottou, editors, NIPS , pages 1593–1600. 2009.

Published in NIPS 2016


Y . Bengio, and L. Bottou, editors, NIPS , pages 1593–1600. 2009.
[28] J.B. Tenenbaum, C. Kemp, T.L. Grifﬁths, and N.D. Goodman. How to grow a mind: Statistics, structure,
and abstraction. Science , 331(6022):1279, 2011.
[29] P Winston and B Horn. The psychology of computer vision , volume 73. McGraw-Hill New York, 1975.
[30] J Wu, I Yildirim, JJ Lim, B Freeman, and J Tenenbaum. Galileo: Perceiving physical object properties by
integrating a physics engine with deep learning. In NIPS , pages 127–135, 2015.
10
A Supplementary material
A.1 Experimental details
A.1.1 Physics engine details
Every simulated trajectory contained the states of the objects in the system on each frame of a sequence of
1000 one-millisecond time steps. The parameters of each system were chosen so that a diverse set of dynamics
could unfold within the trajectory. On each step, the physics engine took as input the current state of the
system, calculated the forces associated with each inter-entity interaction, and applied them to the individual

Published in NIPS 2016


system, calculated the forces associated with each inter-entity interaction, and applied them to the individual
entities as accelerations by dividing by the entity’s mass, parameterized as inverse-mass, a=Fm1in both the
engine and model input. This also allows static objects to be represented as having m1= 0. The previous
positions and velocities, and newly computed accelerations, were input to an Euler integrator to update the
current velocities and positions of the entities. By using a custom engine we were able to have perfect control
over the details of the simulation, and use one engine for all physical domains. It produced trajectory rollouts
that were indistinguishable from those of an off-the-shelf simulation engine, but was highly efﬁcient because it
could perform thousands of runs in parallel on a GPU, allowing millions of simulated steps to be generated each
second.
A.1.2 Physical domains
n-body All objects in n-body systems exerted gravitational forces on each other, which were a function of
the objects’ pairwise distances and masses, giving rise to very complex dynamics that are highly sensitive to
initial conditions. Across simulation runs, the objects’ masses varied, while all other non-dynamic variables
were held constant. The gravitational constant was set to ensure that objects could move across several hundred

Published in NIPS 2016


were held constant. The gravitational constant was set to ensure that objects could move across several hundred
meters within the 1000 step rollout. The training scenes always included 6 objects, and for testing we used 3,
6, and 12 objects. The masses were uniformly sampled from [0:02;9]kg, their shapes were points, and their
initial positions were randomly sampled from all angles, with a distance in [10;100] m. We included two classes
of scenes. The ﬁrst, orbit systems, had one object (the star), initialized at position (0;0), with zero velocity
and a mass of 100kg. The planets’ velocities were initialized such that they would have stable orbits around
the star, if not for their interactions with other planets. The second, non-orbit systems, sampled initial x- and
y-velocity components from [3;3]m/s. The objects would typically begin attracting, and gave rise to complex
semi-periodic behavior.
An n-body system is a highly nonlinear (chaotic) dynamical system, which means they are highly sensitive to
initial conditions and extended predictions of their states are not possible under even small perturbations. The
relations between the objects corresponded to gravitational attraction. Between simulation runs, the masses of

Published in NIPS 2016


initial conditions and extended predictions of their states are not possible under even small perturbations. The
relations between the objects corresponded to gravitational attraction. Between simulation runs, the masses of
the objects were varied, while all other non-dynamic variables were held constant (e.g., gravitational constant) or
were not meaningful (e.g., object scales and shapes). The gravitational force from object itojwas computed as,
Fij=Gmimj(xixj)
kxixjk3, where Gis the gravitational constant. The denominator was clipped so that forces could
not go too high as the distances between objects went to zero. All forces applied to each object were summed to
yield the per-object total forces.
Bouncing balls The bouncing balls domain still had all-to-all object relations–collisions–and any object
could collide with any other, including the walls. But colliding objects are more difﬁcult to simulate than the
gravitational interactions in n-body systems, and our bouncing balls domain also included more variability in
object attributes, such as shape, scale, and mass (as before), as well as relation attributes, such as the coefﬁcient
of restitution. The data distribution was much more challenging: for more than 99% of the time steps, a ball was

Published in NIPS 2016


of restitution. The data distribution was much more challenging: for more than 99% of the time steps, a ball was
not in contact with any others, and its next-step velocity equaled its current velocity. For the remaining steps,
however, collisions caused next-step velocities that was a complex function of its state and the state of the object
it collides with. The training scene contained 6 balls inside a box with 4 walls, and test scenes contained either 3,
6, or 9 balls. The balls’ radii were sampled uniformly from [0:1;0:3]m, and masses from [0:75;1:25]kg. The
walls were static rectangular boxes, positioned so that the horizontal and vertical lengths of area they enclosed
varied independently between [1;3]m. The balls’ initial positions were randomly sampled so that all balls ﬁt
within the box and did not interpenetrate any other object, with initial x- andy-velocity components sampled
uniformly from [5;5]m/s. The restitutions, an attribute of their collision relations, was sampled uniformly
from[0:4;1].
Rigid body collision systems are highly nonlinear (chaotic) dynamical systems, as well. Collision forces were

Published in NIPS 2016


from[0:4;1].
Rigid body collision systems are highly nonlinear (chaotic) dynamical systems, as well. Collision forces were
applied between objects when they began to interpenetrate, using two-step process: collision detection between
all objects, and resolution of detected collisions. A detected collision between two objects meant that their
shapes overlapped and their velocities were causing them to approach each other. To resolve a partially inelastic
collision, the post-collision velocities of each object were computed, and forces appropriate to effect these
velocities on the subsequent time step were then calculated and applied the the objects. This resulted in realistic
bouncing ball trajectories.
11
Table 3: Prediction experiments - Generalization MSEs
n-body Balls String
Model 3 12 3 9 5, 1 30, 1 15, 0 15, 2
Const. vel. 45 166 0.054 0.084 0.0080 0.010 0.020 0.012
IN (within) 0.21 5.2 0.0053 0.0019 0.0030 0.00066 0.0010 0.00097
IN (transfer) 0.052 7.8 0.0053 0.004 0.00052 0.00074 0.0014 0.00097

Published in NIPS 2016


IN (transfer) 0.052 7.8 0.0053 0.004 0.00052 0.00074 0.0014 0.00097
String Our string domain used multiple types of relations (springs and collisions), relation structures that were
more sparse and speciﬁc than all-to-all, and variable external effects. Each scene contained one string, comprised
of point masses connected by springs, and one static, rigid circle positioned below the string. Gravitational
acceleration, varied across simulation runs, was applied so that the string always fell, usually colliding with
the static object. Each training scene contained a string with 15 point masses, and test scenes contained either
5, 15, or 30 mass strings. In training, one of the point masses at the end of the string, chosen at random, was
always held static, as if pinned to the wall, while the other masses were free to move. In the test conditions, we
also included strings that had both ends pinned, and no ends pinned, to evaluate generalization. The string’s
masses were sampled from [0:05;0:15]kg. The springs’ rest lengths were set to 0:2m, spring constants to 100,

Published in NIPS 2016


masses were sampled from [0:05;0:15]kg. The springs’ rest lengths were set to 0:2m, spring constants to 100,
and damping factors to 0:001. The static object’s x-position was sampled from [0:5;0:5]m,y-position was
sampled from [1;0:5]m, and radius from [0:2;0:4]m. The string mass-rigid object coefﬁcient of restitution
was sampled from [0;1]. The gravitational acceleration applied to all non-static objects was sampled from
[30;5]m/s2.
The spring force from object itojwas computed using Hooke’s law as, F=Cs(1L
kxixjk)(xixj),
where Csis the spring constant and Lis the spring’s rest length. A damping factor, which was proportional to
the difference in the objects’ velocities, was also applied. Collision forces between the string’s masses and the
static, rigid object were calculated as described in the bouncing balls section above.
A.2 Results details
The prediction experiment’s generalization performance MSEs are shown Table 3. The URL links for the videos

Published in NIPS 2016


A.2 Results details
The prediction experiment’s generalization performance MSEs are shown Table 3. The URL links for the videos
associated with all still frames in Figure 2 are in Table 1.
12

Published in NIPS 2016


Mask R-CNN
Kaiming He Georgia Gkioxari Piotr Doll ´ar Ross Girshick
Facebook AI Research (FAIR)
Abstract
We present a conceptually simple, ﬂexible, and general
framework for object instance segmentation. Our approach
efﬁciently detects objects in an image while simultaneously
generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster
R-CNN by adding a branch for predicting an object mask in
parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small
overhead to Faster R-CNN, running at 5 fps. Moreover,
Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework.
We show top results in all three tracks of the COCO suite
of challenges, including instance segmentation, boundingbox object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the
COCO 2016 challenge winners. We hope our simple and
effective approach will serve as a solid baseline and help
ease future research in instance-level recognition. Code
has been made available at: https://github.com/

Mask R-CNN

Summary:
We present a conceptually simple, flexible, and general framework for object
instance segmentation. Our approach efficiently detects objects in an image
while simultaneously generating a high-quality segmentation mask for each
instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a
branch for predicting an object mask in parallel with the existing branch for
bounding box recognition. Mask R-CNN is simple to train and adds only a small
overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to
generalize to other tasks, e.g., allowing us to estimate human poses in the
same framework. We show top results in all three tracks of the COCO suite of
challenges, including instance segmentation, bounding-box object detection, and
person keypoint detection. Without bells and whistles, Mask R-CNN outperforms
all existing, single-model entries on every task, including the COCO 2016
challenge winners. We hope our simple and effective approach will serve as a
solid baseline and help ease future research in instance-level recognition.
Code has been made available at: https://github.com/facebookresearch/Detectron

Kaiming He
Georgia Gkioxari
Piotr Dollár
Ross Girshick

open source; appendix on more results


effective approach will serve as a solid baseline and help
ease future research in instance-level recognition. Code
has been made available at: https://github.com/
facebookresearch/Detectron .
1. Introduction
The vision community has rapidly improved object detection and semantic segmentation results over a short period of time. In large part, these advances have been driven
by powerful baseline systems, such as the Fast/Faster RCNN [12, 36] and Fully Convolutional Network (FCN) [30]
frameworks for object detection and semantic segmentation, respectively. These methods are conceptually intuitive
and offer ﬂexibility and robustness, together with fast training and inference time. Our goal in this work is to develop a
comparably enabling framework for instance segmentation .
Instance segmentation is challenging because it requires
the correct detection of all objects in an image while also
precisely segmenting each instance. It therefore combines
elements from the classical computer vision tasks of object detection , where the goal is to classify individual objects and localize each using a bounding box, and semantic
RoIAlignRoIAlignclass
box
convconv
 convconvFigure 1. The Mask R-CNN framework for instance segmentation.
segmentation , where the goal is to classify each pixel into

Mask R-CNN

open source; appendix on more results


box
convconv
 convconvFigure 1. The Mask R-CNN framework for instance segmentation.
segmentation , where the goal is to classify each pixel into
a ﬁxed set of categories without differentiating object instances.1Given this, one might expect a complex method
is required to achieve good results. However, we show that
a surprisingly simple, ﬂexible, and fast system can surpass
prior state-of-the-art instance segmentation results.
Our method, called Mask R-CNN , extends Faster R-CNN
[36] by adding a branch for predicting segmentation masks
on each Region of Interest (RoI), in parallel with the existing branch for classiﬁcation and bounding box regression (Figure 1). The mask branch is a small FCN applied
to each RoI, predicting a segmentation mask in a pixel-topixel manner. Mask R-CNN is simple to implement and
train given the Faster R-CNN framework, which facilitates
a wide range of ﬂexible architecture designs. Additionally,
the mask branch only adds a small computational overhead,
enabling a fast system and rapid experimentation.
In principle Mask R-CNN is an intuitive extension of
Faster R-CNN, yet constructing the mask branch properly

Mask R-CNN

open source; appendix on more results


enabling a fast system and rapid experimentation.
In principle Mask R-CNN is an intuitive extension of
Faster R-CNN, yet constructing the mask branch properly
is critical for good results. Most importantly, Faster RCNN was not designed for pixel-to-pixel alignment between network inputs and outputs. This is most evident in
how RoIPool [18, 12], the de facto core operation for attending to instances, performs coarse spatial quantization
for feature extraction. To ﬁx the misalignment, we propose a simple, quantization-free layer, called RoIAlign , that
faithfully preserves exact spatial locations. Despite being
1Following common terminology, we use object detection to denote
detection via bounding boxes , not masks, and semantic segmentation to
denote per-pixel classiﬁcation without differentiating instances. Yet we
note that instance segmentation is both semantic and a form of detection.
1arXiv:1703.06870v3  [cs.CV]  24 Jan 2018
dining table.96person1.00person1.00 person1.00 person1.00 person1.00person1.00
person1.00person.94
bottle.99
bottle.99
bottle.99
motorcycle1.00motorcycle1.00person1.00
person1.00

Mask R-CNN

open source; appendix on more results


bottle.99
bottle.99
bottle.99
motorcycle1.00motorcycle1.00person1.00
person1.00
person.96person1.00person.83person.96
person.98 person.90person.92person.99person.91
bus.99
person1.00
person1.00person1.00
backpack.93person1.00person.99
person1.00
backpack.99
person.99person.98person.89person.95
person1.00
person1.00
car1.00traffic light .96
person.96truck1.00person.99
car.99person.85
motorcycle.95car.99car.92person.99person1.00traffic light .92 traffic light .84traffic light .95
car.93person.87
person1.00
person1.00umbrella.98
umbrella.98
backpack1.00
handba g.96
elephant1.00person1.00person1.00 person.99
sheep1. 00person1.00
sheep.99sheep.91 sheep1. 00
sheep.99sheep.99sheep.95person.99

Mask R-CNN

open source; appendix on more results


sheep1. 00person1.00
sheep.99sheep.91 sheep1. 00
sheep.99sheep.99sheep.95person.99
sheep1. 00sheep.96sheep.99sheep.99
sheep.96
sheep.96sheep.96sheep.86
sheep.82sheep.93
dining table.99
chair.99chair.90
chair.99chair.98
chair.96chair.86chair.99
bowl.81chair.96tv.99
bottle.99
wine glass.99wine glass1.00bowl.85
knife.83wine glass1.00 wine glass.93wine glass.97
fork.95Figure 2. Mask R-CNN results on the COCO test set. These results are based on ResNet-101 [19], achieving a mask AP of 35.7 and
running at 5 fps. Masks are shown in color, and bounding box, category, and conﬁdences are also shown.
a seemingly minor change, RoIAlign has a large impact: it
improves mask accuracy by relative 10% to 50%, showing
bigger gains under stricter localization metrics. Second, we
found it essential to decouple mask and class prediction: we
predict a binary mask for each class independently, without

Mask R-CNN

open source; appendix on more results


bigger gains under stricter localization metrics. Second, we
found it essential to decouple mask and class prediction: we
predict a binary mask for each class independently, without
competition among classes, and rely on the network’s RoI
classiﬁcation branch to predict the category. In contrast,
FCNs usually perform per-pixel multi-class categorization,
which couples segmentation and classiﬁcation, and based
on our experiments works poorly for instance segmentation.
Without bells and whistles, Mask R-CNN surpasses all
previous state-of-the-art single-model results on the COCO
instance segmentation task [28], including the heavilyengineered entries from the 2016 competition winner. As
a by-product, our method also excels on the COCO object
detection task. In ablation experiments, we evaluate multiple basic instantiations, which allows us to demonstrate its
robustness and analyze the effects of core factors.
Our models can run at about 200ms per frame on a GPU,
and training on COCO takes one to two days on a single
8-GPU machine. We believe the fast train and test speeds,
together with the framework’s ﬂexibility and accuracy, will
beneﬁt and ease future research on instance segmentation.

Mask R-CNN

open source; appendix on more results


together with the framework’s ﬂexibility and accuracy, will
beneﬁt and ease future research on instance segmentation.
Finally, we showcase the generality of our framework
via the task of human pose estimation on the COCO keypoint dataset [28]. By viewing each keypoint as a one-hot
binary mask, with minimal modiﬁcation Mask R-CNN can
be applied to detect instance-speciﬁc poses. Mask R-CNN
surpasses the winner of the 2016 COCO keypoint competition, and at the same time runs at 5 fps. Mask R-CNN,
therefore, can be seen more broadly as a ﬂexible framework
forinstance-level recognition and can be readily extended
to more complex tasks.
We have released code to facilitate future research.2. Related Work
R-CNN: The Region-based CNN (R-CNN) approach [13]
to bounding-box object detection is to attend to a manageable number of candidate object regions [42, 20] and evaluate convolutional networks [25, 24] independently on each
RoI. R-CNN was extended [18, 12] to allow attending to
RoIs on feature maps using RoIPool, leading to fast speed
and better accuracy. Faster R-CNN [36] advanced this

Mask R-CNN

open source; appendix on more results


RoIs on feature maps using RoIPool, leading to fast speed
and better accuracy. Faster R-CNN [36] advanced this
stream by learning the attention mechanism with a Region
Proposal Network (RPN). Faster R-CNN is ﬂexible and robust to many follow-up improvements ( e.g., [38, 27, 21]),
and is the current leading framework in several benchmarks.
Instance Segmentation: Driven by the effectiveness of RCNN, many approaches to instance segmentation are based
onsegment proposals . Earlier methods [13, 15, 16, 9] resorted to bottom-up segments [42, 2]. DeepMask [33] and
following works [34, 8] learn to propose segment candidates, which are then classiﬁed by Fast R-CNN. In these
methods, segmentation precedes recognition, which is slow
and less accurate. Likewise, Dai et al. [10] proposed a complex multiple-stage cascade that predicts segment proposals
from bounding-box proposals, followed by classiﬁcation.
Instead, our method is based on parallel prediction of masks
and class labels, which is simpler and more ﬂexible.
Most recently, Li et al. [26] combined the segment proposal system in [8] and object detection system in [11] for

Mask R-CNN

open source; appendix on more results


Most recently, Li et al. [26] combined the segment proposal system in [8] and object detection system in [11] for
“fully convolutional instance segmentation” (FCIS). The
common idea in [8, 11, 26] is to predict a set of positionsensitive output channels fully convolutionally. These
channels simultaneously address object classes, boxes, and
masks, making the system fast. But FCIS exhibits systematic errors on overlapping instances and creates spurious
edges (Figure 6), showing that it is challenged by the fundamental difﬁculties of segmenting instances.
2
Another family of solutions [23, 4, 3, 29] to instance segmentation are driven by the success of semantic segmentation. Starting from per-pixel classiﬁcation results ( e.g.,
FCN outputs), these methods attempt to cut the pixels of
the same category into different instances. In contrast to the
segmentation-ﬁrst strategy of these methods, Mask R-CNN
is based on an instance-ﬁrst strategy. We expect a deeper incorporation of both strategies will be studied in the future.
3. Mask R-CNN
Mask R-CNN is conceptually simple: Faster R-CNN has
two outputs for each candidate object, a class label and a

Mask R-CNN

open source; appendix on more results


3. Mask R-CNN
Mask R-CNN is conceptually simple: Faster R-CNN has
two outputs for each candidate object, a class label and a
bounding-box offset; to this we add a third branch that outputs the object mask. Mask R-CNN is thus a natural and intuitive idea. But the additional mask output is distinct from
the class and box outputs, requiring extraction of much ﬁner
spatial layout of an object. Next, we introduce the key elements of Mask R-CNN, including pixel-to-pixel alignment,
which is the main missing piece of Fast/Faster R-CNN.
Faster R-CNN: We begin by brieﬂy reviewing the Faster
R-CNN detector [36]. Faster R-CNN consists of two stages.
The ﬁrst stage, called a Region Proposal Network (RPN),
proposes candidate object bounding boxes. The second
stage, which is in essence Fast R-CNN [12], extracts features using RoIPool from each candidate box and performs
classiﬁcation and bounding-box regression. The features
used by both stages can be shared for faster inference. We
refer readers to [21] for latest, comprehensive comparisons
between Faster R-CNN and other frameworks.
Mask R-CNN: Mask R-CNN adopts the same two-stage

Mask R-CNN

open source; appendix on more results


refer readers to [21] for latest, comprehensive comparisons
between Faster R-CNN and other frameworks.
Mask R-CNN: Mask R-CNN adopts the same two-stage
procedure, with an identical ﬁrst stage (which is RPN). In
the second stage, in parallel to predicting the class and box
offset, Mask R-CNN also outputs a binary mask for each
RoI. This is in contrast to most recent systems, where classiﬁcation depends on mask predictions ( e.g. [33, 10, 26]).
Our approach follows the spirit of Fast R-CNN [12] that
applies bounding-box classiﬁcation and regression in parallel (which turned out to largely simplify the multi-stage
pipeline of original R-CNN [13]).
Formally, during training, we deﬁne a multi-task loss on
each sampled RoI as L=Lcls+Lbox+Lmask . The classiﬁcation loss Lclsand bounding-box loss Lboxare identical as those deﬁned in [12]. The mask branch has a Km2dimensional output for each RoI, which encodes Kbinary
masks of resolution mm, one for each of the Kclasses.

Mask R-CNN

open source; appendix on more results


masks of resolution mm, one for each of the Kclasses.
To this we apply a per-pixel sigmoid, and deﬁne Lmask as
the average binary cross-entropy loss. For an RoI associated
with ground-truth class k,Lmask is only deﬁned on the k-th
mask (other mask outputs do not contribute to the loss).
Our deﬁnition of Lmask allows the network to generate
masks for every class without competition among classes;
we rely on the dedicated classiﬁcation branch to predict the
Figure 3. RoIAlign: The dashed grid represents a feature map, the solid lines an RoI
(with 22 bins in this example), and the dots
the 4 sampling points in each bin. RoIAlign
computes the value of each sampling point
by bilinear interpolation from the nearby grid
points on the feature map. No quantization is
performed on any coordinates involved in the
RoI, its bins, or the sampling points.
class label used to select the output mask. This decouples
mask and class prediction. This is different from common
practice when applying FCNs [30] to semantic segmentation, which typically uses a per-pixel softmax and a multinomial cross-entropy loss. In that case, masks across classes

Mask R-CNN

open source; appendix on more results


practice when applying FCNs [30] to semantic segmentation, which typically uses a per-pixel softmax and a multinomial cross-entropy loss. In that case, masks across classes
compete; in our case, with a per-pixel sigmoid and a binary
loss, they do not. We show by experiments that this formulation is key for good instance segmentation results.
Mask Representation: A mask encodes an input object’s
spatial layout. Thus, unlike class labels or box offsets
that are inevitably collapsed into short output vectors by
fully-connected ( fc) layers, extracting the spatial structure
of masks can be addressed naturally by the pixel-to-pixel
correspondence provided by convolutions.
Speciﬁcally, we predict an mmmask from each RoI
using an FCN [30]. This allows each layer in the mask
branch to maintain the explicit mmobject spatial layout without collapsing it into a vector representation that
lacks spatial dimensions. Unlike previous methods that resort to fclayers for mask prediction [33, 34, 10], our fully
convolutional representation requires fewer parameters, and
is more accurate as demonstrated by experiments.
This pixel-to-pixel behavior requires our RoI features,
which themselves are small feature maps, to be well aligned

Mask R-CNN

open source; appendix on more results


is more accurate as demonstrated by experiments.
This pixel-to-pixel behavior requires our RoI features,
which themselves are small feature maps, to be well aligned
to faithfully preserve the explicit per-pixel spatial correspondence. This motivated us to develop the following
RoIAlign layer that plays a key role in mask prediction.
RoIAlign: RoIPool [12] is a standard operation for extracting a small feature map ( e.g., 77) from each RoI. RoIPool
ﬁrstquantizes a ﬂoating-number RoI to the discrete granularity of the feature map, this quantized RoI is then subdivided into spatial bins which are themselves quantized, and
ﬁnally feature values covered by each bin are aggregated
(usually by max pooling). Quantization is performed, e.g.,
on a continuous coordinate xby computing [x=16], where
16 is a feature map stride and []is rounding; likewise, quantization is performed when dividing into bins ( e.g., 77).
These quantizations introduce misalignments between the
RoI and the extracted features. While this may not impact
classiﬁcation, which is robust to small translations, it has a
large negative effect on predicting pixel-accurate masks.

Mask R-CNN

open source; appendix on more results


classiﬁcation, which is robust to small translations, it has a
large negative effect on predicting pixel-accurate masks.
To address this, we propose an RoIAlign layer that removes the harsh quantization of RoIPool, properly aligning
the extracted features with the input. Our proposed change
is simple: we avoid any quantization of the RoI boundaries
3
or bins ( i.e., we use x=16instead of [x=16]). We use bilinear interpolation [22] to compute the exact values of the
input features at four regularly sampled locations in each
RoI bin, and aggregate the result (using max or average),
see Figure 3 for details. We note that the results are not sensitive to the exact sampling locations, or how many points
are sampled, as long as no quantization is performed.
RoIAlign leads to large improvements as we show in
x4.2. We also compare to the RoIWarp operation proposed
in [10]. Unlike RoIAlign, RoIWarp overlooked the alignment issue and was implemented in [10] as quantizing RoI
just like RoIPool. So even though RoIWarp also adopts
bilinear resampling motivated by [22], it performs on par

Mask R-CNN

open source; appendix on more results


just like RoIPool. So even though RoIWarp also adopts
bilinear resampling motivated by [22], it performs on par
with RoIPool as shown by experiments (more details in Table 2c), demonstrating the crucial role of alignment.
Network Architecture: To demonstrate the generality of
our approach, we instantiate Mask R-CNN with multiple
architectures. For clarity, we differentiate between: (i) the
convolutional backbone architecture used for feature extraction over an entire image, and (ii) the network head
for bounding-box recognition (classiﬁcation and regression)
and mask prediction that is applied separately to each RoI.
We denote the backbone architecture using the nomenclature network-depth-features . We evaluate ResNet [19]
and ResNeXt [45] networks of depth 50 or 101 layers. The
original implementation of Faster R-CNN with ResNets
[19] extracted features from the ﬁnal convolutional layer
of the 4-th stage, which we call C4. This backbone with
ResNet-50, for example, is denoted by ResNet-50-C4. This
is a common choice used in [19, 10, 21, 39].

Mask R-CNN

open source; appendix on more results


ResNet-50, for example, is denoted by ResNet-50-C4. This
is a common choice used in [19, 10, 21, 39].
We also explore another more effective backbone recently proposed by Lin et al. [27], called a Feature Pyramid Network (FPN). FPN uses a top-down architecture with
lateral connections to build an in-network feature pyramid
from a single-scale input. Faster R-CNN with an FPN backbone extracts RoI features from different levels of the feature pyramid according to their scale, but otherwise the
rest of the approach is similar to vanilla ResNet. Using a
ResNet-FPN backbone for feature extraction with Mask RCNN gives excellent gains in both accuracy and speed. For
further details on FPN, we refer readers to [27].
For the network head we closely follow architectures
presented in previous work to which we add a fully convolutional mask prediction branch. Speciﬁcally, we extend the Faster R-CNN box heads from the ResNet [19]
and FPN [27] papers. Details are shown in Figure 4. The
head on the ResNet-C4 backbone includes the 5-th stage of
ResNet (namely, the 9-layer ‘res5’ [19]), which is computeintensive. For FPN, the backbone already includes res5 and

Mask R-CNN

open source; appendix on more results


ResNet (namely, the 9-layer ‘res5’ [19]), which is computeintensive. For FPN, the backbone already includes res5 and
thus allows for a more efﬁcient head that uses fewer ﬁlters.
We note that our mask branches have a straightforward
structure. More complex designs have the potential to improve performance but are not the focus of this work.
ave
RoI
RoI14×14
×2567×7
×256
14×14
×2561024
28×28
×2561024
mask14×14
×256class
box2048RoI res57×7
×10247×7
×2048
×4class
box
14×14
×80
mask28×28
×80Faster R-CNN
w/ ResNet [19]Faster R-CNN
w/ FPN [27]
Figure 4. Head Architecture : We extend two existing Faster RCNN heads [19, 27]. Left/Right panels show the heads for the
ResNet C4 and FPN backbones, from [19] and [27], respectively,
to which a mask branch is added. Numbers denote spatial resolution and channels. Arrows denote either conv, deconv, or fclayers
as can be inferred from context (conv preserves spatial dimension

Mask R-CNN

open source; appendix on more results


to which a mask branch is added. Numbers denote spatial resolution and channels. Arrows denote either conv, deconv, or fclayers
as can be inferred from context (conv preserves spatial dimension
while deconv increases it). All convs are 3 3, except the output
conv which is 1 1, deconvs are 2 2 with stride 2, and we use
ReLU [31] in hidden layers. Left: ‘res5’ denotes ResNet’s ﬁfth
stage, which for simplicity we altered so that the ﬁrst conv operates on a 7 7 RoI with stride 1 (instead of 14 14 / stride 2 as in
[19]). Right : ‘4’ denotes a stack of four consecutive convs.
3.1. Implementation Details
We set hyper-parameters following existing Fast/Faster
R-CNN work [12, 36, 27]. Although these decisions were
made for object detection in original papers [12, 36, 27], we
found our instance segmentation system is robust to them.
Training: As in Fast R-CNN, an RoI is considered positive
if it has IoU with a ground-truth box of at least 0.5 and
negative otherwise. The mask loss Lmask is deﬁned only on
positive RoIs. The mask target is the intersection between

Mask R-CNN

open source; appendix on more results


negative otherwise. The mask loss Lmask is deﬁned only on
positive RoIs. The mask target is the intersection between
an RoI and its associated ground-truth mask.
We adopt image-centric training [12]. Images are resized
such that their scale (shorter edge) is 800 pixels [27]. Each
mini-batch has 2 images per GPU and each image has N
sampled RoIs, with a ratio of 1:3 of positive to negatives
[12]. Nis 64 for the C4 backbone (as in [12, 36]) and 512
for FPN (as in [27]). We train on 8 GPUs (so effective minibatch size is 16) for 160k iterations, with a learning rate of
0.02 which is decreased by 10 at the 120k iteration. We
use a weight decay of 0.0001 and momentum of 0.9. With
ResNeXt [45], we train with 1 image per GPU and the same
number of iterations, with a starting learning rate of 0.01.
The RPN anchors span 5 scales and 3 aspect ratios, following [27]. For convenient ablation, RPN is trained separately and does not share features with Mask R-CNN, unless speciﬁed. For every entry in this paper, RPN and Mask
R-CNN have the same backbones and so they are shareable.

Mask R-CNN

open source; appendix on more results


R-CNN have the same backbones and so they are shareable.
Inference: At test time, the proposal number is 300 for the
C4 backbone (as in [36]) and 1000 for FPN (as in [27]). We
run the box prediction branch on these proposals, followed
by non-maximum suppression [14]. The mask branch is
then applied to the highest scoring 100 detection boxes. Although this differs from the parallel computation used in
training, it speeds up inference and improves accuracy (due
to the use of fewer, more accurate RoIs). The mask branch
4
horse1.00horse1.00horse1.00
bus1.00bus1.00
car.98truck.88
car.93
car.78car.98
car.91 car.96car.99
car.94car.99car.98 truck.86
car.99car.95car1.00car.93 car.98car.95
car.97car.87
car.99
car.82car.78car.93
car.95
car.97
person.99traffic light .73
person1.00
person.99person.95person.93
person.93
person1.00person.98
skateboard.82
suitcase1.00
suitcase.99suitcase.96suitcase1.00

Mask R-CNN

open source; appendix on more results


person.93
person1.00person.98
skateboard.82
suitcase1.00
suitcase.99suitcase.96suitcase1.00
suitcase.93suitcase.98
suitcase.88suitcase.72stop sign.88
person1.00 person1.00
person1.00person1.00
person.99person.99
bench.76skateboard.91
skateboard.83handba g.81
surfboard1.00person1.00person1.00 surfboard1.00person1.00person.98
surfboard1.00person1.00
surfboard.98surfboard1.00person.91
person.74
person1.00person1.00
person1.00person1.00person1.00 person1.00 person.98person.99
person1.00person.99 umbrella1.00
person.95umbrella.99umbrella.97umbrella.97
umbrella.96
umbrella1.00
backpack.96umbrella.98
backpack.95person.80
backpack.98
bicycle.93umbrella.89person.89handba g.97
handba g.85

Mask R-CNN

open source; appendix on more results


backpack.95person.80
backpack.98
bicycle.93umbrella.89person.89handba g.97
handba g.85
person1.00person1.00 person1.00person1.00person1.00person1.00
motorcycle.72kite.89
person.99kite.95
person.99
person1.00person.81person.72kite.93
person.89kite1.00
person.98
person1.00kite.84kite.97
person.80
handba g.80person.99kite.82
person.98 person.96kite.98
person.99person.82kite.81
person.95 person.84kite.98kite.72
kite.99kite.84kite.99
person.94person.72person.98kite.95
person.98 person.77kite.73
person.78 person.71 person.87kite.88kite.88
person.94kite.86kite.89
zebra.99
zebra1.00zebra1.00
zebra.99zebra1.00zebra.96
zebra.74
zebra.96zebra.99zebra.90

Mask R-CNN

open source; appendix on more results


zebra.99zebra1.00zebra.96
zebra.74
zebra.96zebra.99zebra.90
zebra.88zebra.76
dining table.91dining table.78
chair.97person.99
person.86
chair.94chair.98person.95
chair.95person.97
chair.92chair.99person.97
person.99person.94 person.99person.87
person.99
chair.83person.94person.99person.98
chair.87chair.95person.97
person.96
chair.99person.86person.89
chair.89
wine glass.93person.98person.88person.97
person.88person.88
person.91 chair.96person.95
person.77person.92
wine glass.94cup.83
wine glass.94
wine glass.83
cup.91chair.85 dining table.96
wine glass.91person.96
cup.98person.83
dining table.75
cup.96person.72
wine glass.80chair.98person.81person.82
dining table.81
chair.85chair.78
cup.75person.77
cup.71wine glass.80cup.79cup.93
cup.71

Mask R-CNN

open source; appendix on more results


dining table.81
chair.85chair.78
cup.75person.77
cup.71wine glass.80cup.79cup.93
cup.71
person.99person.99
person1.00person1.00
frisbee1.00
person.80person.82
elephant1.00elephant1.00elephant1.00
elephant.97
elephant.99
person1.00person1.00
dining table.95person1.00person.88
wine glass1.00bottle.97
wine glass1.00wine glass.99tv.98 tv.84
person1.00
bench.97person.98person1.00person1.00
handba g.73person.86 potted plant.92
bird.93person.76person.98person.78 person.78 backpack.88handba g.91
cell phone.77 clock.73
person.99person1.00person.98
person1.00person1.00 person1.00person.99
person.99 person.99 person1.00 person1.00person.98 person.99
handba g.88person1.00 person.98person.92
handba g.99person.97
person.95
handba g.88traffic light .99
person.95

Mask R-CNN

open source; appendix on more results


handba g.99person.97
person.95
handba g.88traffic light .99
person.95
person.87person.95traffic light .87
traffic light .71
person.80person.95 person.95 person.73person.74
tie.85
car.99
car.86car.97car1.00 car.95car.97traffic light 1.00traffic light .99
car.99person.99car.95
car.97 car.98car.98
car.91
car1.00car.96car.96
bicycle.86car.97car.97
car.97car.94car.95
car.94car.81
person.87
parking meter.98car.89
donut1.00donut.90
donut.88donut.81
donut.95
donut.96donut1.00 donut.98
donut.99donut.94donut.97donut.99
donut.98donut1.00
donut.95donut1.00
donut.98donut.98donut.99
donut.96
donut.89donut.96donut.95donut.98donut.89
donut.93donut.95

Mask R-CNN

open source; appendix on more results


donut.96
donut.89donut.96donut.95donut.98donut.89
donut.93donut.95
donut.90donut.89
donut.89donut.89
donut.86donut.86
person1.00person1.00 person1.00
person1.00person1.00person1.00
person1.00
dog1.00baseball bat.99
baseball bat.85
baseball bat.98
truck.92
truck.99truck.96truck.99truck.97
bus.99truck.93 bus.90
person1.00person1.00horse.77horse.99
cow.93person.96person1.00
person.99horse.97
person.98 person.97person.98
person.96
person1.00
tennis racket1.00chair.73person.90person.77
person.97
person.81person.87
person.71 person.96 person.99 person.98 person.94chair.97
chair.80
chair.71chair.94chair.92chair.99chair.93
chair.99
chair.91 chair.81 chair.98 chair.83chair.81chair.81
chair.93
sports ball.99

Mask R-CNN

open source; appendix on more results


chair.99
chair.91 chair.81 chair.98 chair.83chair.81chair.81
chair.93
sports ball.99
person1.00
couch.82person1.00
person.99person1.00person1.00person1.00 person.99skateboard.99
person.90person.98person.99person.91
person.99person1.00
person.80
skateboard.98Figure 5. More results of Mask R-CNN on COCO test images, using ResNet-101-FPN and running at 5 fps, with 35.7 mask AP (Table 1).
backbone AP AP 50 AP75 APS APM APL
MNC [10] ResNet-101-C4 24.6 44.3 24.8 4.7 25.9 43.6
FCIS [26] +OHEM ResNet-101-C5-dilated 29.2 49.5 - 7.1 31.3 50.0
FCIS+++ [26] +OHEM ResNet-101-C5-dilated 33.6 54.5 - - - Mask R-CNN ResNet-101-C4 33.1 54.9 34.8 12.1 35.6 51.1

Mask R-CNN

open source; appendix on more results


Mask R-CNN ResNet-101-FPN 35.7 58.0 37.8 15.5 38.1 52.4
Mask R-CNN ResNeXt-101-FPN 37.1 60.0 39.4 16.9 39.9 53.5
Table 1. Instance segmentation mask AP on COCO test-dev . MNC [10] and FCIS [26] are the winners of the COCO 2015 and 2016
segmentation challenges, respectively. Without bells and whistles, Mask R-CNN outperforms the more complex FCIS+++, which includes
multi-scale train/test, horizontal ﬂip test, and OHEM [38]. All entries are single-model results.
can predict Kmasks per RoI, but we only use the k-th mask,
where kis the predicted class by the classiﬁcation branch.
Themmﬂoating-number mask output is then resized to
the RoI size, and binarized at a threshold of 0.5.
Note that since we only compute masks on the top 100
detection boxes, Mask R-CNN adds a small overhead to its
Faster R-CNN counterpart ( e.g.,20% on typical models).
4. Experiments: Instance Segmentation

Mask R-CNN

open source; appendix on more results


Faster R-CNN counterpart ( e.g.,20% on typical models).
4. Experiments: Instance Segmentation
We perform a thorough comparison of Mask R-CNN to
the state of the art along with comprehensive ablations on
the COCO dataset [28]. We report the standard COCO metrics including AP (averaged over IoU thresholds), AP 50,
AP75, and AP S, AP M, AP L(AP at different scales). Unless noted, AP is evaluating using mask IoU. As in previous
work [5, 27], we train using the union of 80k train images
and a 35k subset of val images ( trainval35k ), and report ablations on the remaining 5k val images ( minival ).
We also report results on test-dev [28].4.1. Main Results
We compare Mask R-CNN to the state-of-the-art methods in instance segmentation in Table 1. All instantiations of our model outperform baseline variants of previous state-of-the-art models. This includes MNC [10]
and FCIS [26], the winners of the COCO 2015 and 2016
segmentation challenges, respectively. Without bells and
whistles, Mask R-CNN with ResNet-101-FPN backbone
outperforms FCIS+++ [26], which includes multi-scale

Mask R-CNN

open source; appendix on more results


whistles, Mask R-CNN with ResNet-101-FPN backbone
outperforms FCIS+++ [26], which includes multi-scale
train/test, horizontal ﬂip test, and online hard example mining (OHEM) [38]. While outside the scope of this work, we
expect many such improvements to be applicable to ours.
Mask R-CNN outputs are visualized in Figures 2 and 5.
Mask R-CNN achieves good results even under challenging conditions. In Figure 6 we compare our Mask R-CNN
baseline and FCIS+++ [26]. FCIS+++ exhibits systematic
artifacts on overlapping instances, suggesting that it is challenged by the fundamental difﬁculty of instance segmentation. Mask R-CNN shows no such artifacts.
5
person1.00
person1.00
person1.00person1.00umbrella1.00 umbrella.99
car.99car.93
giraffe1.00 giraffe1.00
person1.00person1.00person1.00person1.00
person.95
sports ball1.00sports ball.98person1.00
person1.00
person1.00
tie.95
tie1.00
FCIS Mask R-CNN
train1.00
train.99
train.80
person1.00 person1.00person1.00

Mask R-CNN

open source; appendix on more results


tie1.00
FCIS Mask R-CNN
train1.00
train.99
train.80
person1.00 person1.00person1.00
person1.00person1.00 person1.00
skateboard.98person.99 person.99
skateboard.99handba g.93
Figure 6. FCIS+++ [26] (top) vs. Mask R-CNN (bottom, ResNet-101-FPN). FCIS exhibits systematic artifacts on overlapping objects.
net-depth-features AP AP 50 AP75
ResNet-50-C4 30.3 51.2 31.5
ResNet-101-C4 32.7 54.2 34.3
ResNet-50-FPN 33.6 55.2 35.3
ResNet-101-FPN 35.4 57.3 37.5
ResNeXt-101-FPN 36.7 59.5 38.9
(a)Backbone Architecture : Better backbones bring expected gains: deeper networks
do better, FPN outperforms C4 features, and
ResNeXt improves on ResNet.AP AP 50 AP75
softmax 24.8 44.1 25.1
sigmoid 30.3 51.2 31.5
+5.5 +7.1 +6.4

Mask R-CNN

open source; appendix on more results


softmax 24.8 44.1 25.1
sigmoid 30.3 51.2 31.5
+5.5 +7.1 +6.4
(b)Multinomial vs. Independent Masks
(ResNet-50-C4): Decoupling via perclass binary masks (sigmoid) gives large
gains over multinomial masks (softmax).align? bilinear? agg. AP AP 50 AP75
RoIPool [12] max 26.9 48.8 26.4
RoIWarp [10]X max 27.2 49.2 27.1
X ave 27.1 48.9 27.1
RoIAlignX X max 30.2 51.0 31.8
X X ave 30.3 51.2 31.5
(c)RoIAlign (ResNet-50-C4): Mask results with various RoI
layers. Our RoIAlign layer improves AP by 3 points and
AP75by5 points. Using proper alignment is the only factor that contributes to the large gap between RoI layers.
AP AP 50 AP75 APbbAPbb
50APbb
75
RoIPool 23.6 46.5 21.6 28.2 52.7 26.9

Mask R-CNN

open source; appendix on more results


AP AP 50 AP75 APbbAPbb
50APbb
75
RoIPool 23.6 46.5 21.6 28.2 52.7 26.9
RoIAlign 30.9 51.8 32.1 34.0 55.3 36.4
+7.3 + 5.3 +10.5 +5.8 +2.6 +9.5
(d)RoIAlign (ResNet-50- C5,stride 32 ): Mask-level and box-level
AP using large-stride features. Misalignments are more severe than
with stride-16 features (Table 2c), resulting in big accuracy gaps.mask branch AP AP 50 AP75
MLP fc: 1024!1024!8028231.5 53.7 32.8
MLP fc: 1024!1024!1024!8028231.5 54.0 32.6
FCN conv: 256!256!256!256!256!80 33.6 55.2 35.3
(e)Mask Branch (ResNet-50-FPN): Fully convolutional networks (FCN) vs.
multi-layer perceptrons (MLP, fully-connected) for mask prediction. FCNs improve results as they take advantage of explicitly encoding spatial layout.

Mask R-CNN

open source; appendix on more results


multi-layer perceptrons (MLP, fully-connected) for mask prediction. FCNs improve results as they take advantage of explicitly encoding spatial layout.
Table 2. Ablations . We train on trainval35k , test on minival , and report mask AP unless otherwise noted.
4.2. Ablation Experiments
We run a number of ablations to analyze Mask R-CNN.
Results are shown in Table 2 and discussed in detail next.
Architecture: Table 2a shows Mask R-CNN with various
backbones. It beneﬁts from deeper networks (50 vs. 101)
and advanced designs including FPN and ResNeXt. We
note that notall frameworks automatically beneﬁt from
deeper or advanced networks (see benchmarking in [21]).
Multinomial vs. Independent Masks: Mask R-CNN decouples mask and class prediction: as the existing box
branch predicts the class label, we generate a mask for each
class without competition among classes (by a per-pixel sigmoid and a binary loss). In Table 2b, we compare this to
using a per-pixel softmax and a multinomial loss (as commonly used in FCN [30]). This alternative couples the tasks
of mask and class prediction, and results in a severe loss

Mask R-CNN

open source; appendix on more results


of mask and class prediction, and results in a severe loss
in mask AP (5.5 points). This suggests that once the instance has been classiﬁed as a whole (by the box branch),
it is sufﬁcient to predict a binary mask without concern for
the categories, which makes the model easier to train.
Class-Speciﬁc vs. Class-Agnostic Masks: Our default instantiation predicts class-speciﬁc masks, i.e., one mmmask per class. Interestingly, Mask R-CNN with classagnostic masks ( i.e., predicting a single mmoutput regardless of class) is nearly as effective: it has 29.7 mask AP
vs. 30.3 for the class-speciﬁc counterpart on ResNet-50-C4.
This further highlights the division of labor in our approach
which largely decouples classiﬁcation and segmentation.
RoIAlign: An evaluation of our proposed RoIAlign layer is
shown in Table 2c. For this experiment we use the ResNet50-C4 backbone, which has stride 16. RoIAlign improves
AP by about 3 points over RoIPool, with much of the gain
coming at high IoU (AP 75). RoIAlign is insensitive to

Mask R-CNN

open source; appendix on more results


AP by about 3 points over RoIPool, with much of the gain
coming at high IoU (AP 75). RoIAlign is insensitive to
max/average pool; we use average in the rest of the paper.
Additionally, we compare with RoIWarp proposed in
MNC [10] that also adopt bilinear sampling. As discussed
inx3, RoIWarp still quantizes the RoI, losing alignment
with the input. As can be seen in Table 2c, RoIWarp performs on par with RoIPool and much worse than RoIAlign.
This highlights that proper alignment is key.
We also evaluate RoIAlign with a ResNet-50-C5 backbone, which has an even larger stride of 32 pixels. We use
the same head as in Figure 4 (right), as the res5 head is not
applicable. Table 2d shows that RoIAlign improves mask
AP by a massive 7.3 points, and mask AP 75by 10.5 points
6
backbone APbbAPbb
50APbb
75APbb
SAPbb
MAPbb
L
Faster R-CNN+++ [19] ResNet-101-C4 34.9 55.7 37.4 15.6 38.7 50.9

Mask R-CNN

open source; appendix on more results


Mask R-CNN

open source; appendix on more results


Mask R-CNN ResNeXt-101-FPN 39.8 62.3 43.4 22.1 43.2 51.2
Table 3. Object detection single-model results (bounding box AP), vs. state-of-the-art on test-dev . Mask R-CNN using ResNet-101FPN outperforms the base variants of all previous state-of-the-art models (the mask output is ignored in these experiments). The gains of
Mask R-CNN over [27] come from using RoIAlign (+1.1 APbb), multitask training (+0.9 APbb), and ResNeXt-101 (+1.6 APbb).
(50% relative improvement ). Moreover, we note that with
RoIAlign, using stride-32 C5 features (30.9 AP) is more accurate than using stride-16 C4 features (30.3 AP, Table 2c).
RoIAlign largely resolves the long-standing challenge of
using large-stride features for detection and segmentation.
Finally, RoIAlign shows a gain of 1.5 mask AP and 0.5
box AP when used with FPN, which has ﬁner multi-level
strides. For keypoint detection that requires ﬁner alignment,
RoIAlign shows large gains even with FPN (Table 6).

Mask R-CNN

open source; appendix on more results


strides. For keypoint detection that requires ﬁner alignment,
RoIAlign shows large gains even with FPN (Table 6).
Mask Branch: Segmentation is a pixel-to-pixel task and
we exploit the spatial layout of masks by using an FCN.
In Table 2e, we compare multi-layer perceptrons (MLP)
and FCNs, using a ResNet-50-FPN backbone. Using FCNs
gives a 2.1 mask AP gain over MLPs. We note that we
choose this backbone so that the conv layers of the FCN
head are not pre-trained, for a fair comparison with MLP.
4.3. Bounding Box Detection Results
We compare Mask R-CNN to the state-of-the-art COCO
bounding-box object detection in Table 3. For this result,
even though the full Mask R-CNN model is trained, only
the classiﬁcation and box outputs are used at inference (the

Mask R-CNN

open source; appendix on more results


even though the full Mask R-CNN model is trained, only
the classiﬁcation and box outputs are used at inference (the
mask output is ignored). Mask R-CNN using ResNet-101FPN outperforms the base variants of all previous state-ofthe-art models, including the single-model variant of GRMI [21], the winner of the COCO 2016 Detection Challenge. Using ResNeXt-101-FPN, Mask R-CNN further improves results, with a margin of 3.0 points box AP over
the best previous single model entry from [39] (which used
Inception-ResNet-v2-TDM).
As a further comparison, we trained a version of Mask
R-CNN but without the mask branch, denoted by “Faster
R-CNN, RoIAlign” in Table 3. This model performs better
than the model presented in [27] due to RoIAlign. On the
other hand, it is 0.9 points box AP lower than Mask R-CNN.
This gap of Mask R-CNN on box detection is therefore due
solely to the beneﬁts of multi-task training.
Lastly, we note that Mask R-CNN attains a small gap
between its mask and box AP: e.g., 2.7 points between 37.1

Mask R-CNN

open source; appendix on more results


Lastly, we note that Mask R-CNN attains a small gap
between its mask and box AP: e.g., 2.7 points between 37.1
(mask, Table 1) and 39.8 (box, Table 3). This indicates that
our approach largely closes the gap between object detection and the more challenging instance segmentation task.4.4. Timing
Inference: We train a ResNet-101-FPN model that shares
features between the RPN and Mask R-CNN stages, following the 4-step training of Faster R-CNN [36]. This model
runs at 195ms per image on an Nvidia Tesla M40 GPU (plus
15ms CPU time resizing the outputs to the original resolution), and achieves statistically the same mask AP as the
unshared one. We also report that the ResNet-101-C4 variant takes 400ms as it has a heavier box head (Figure 4), so
we do not recommend using the C4 variant in practice.
Although Mask R-CNN is fast, we note that our design
is not optimized for speed, and better speed/accuracy tradeoffs could be achieved [21], e.g., by varying image sizes and
proposal numbers, which is beyond the scope of this paper.
Training: Mask R-CNN is also fast to train. Training with

Mask R-CNN

open source; appendix on more results


proposal numbers, which is beyond the scope of this paper.
Training: Mask R-CNN is also fast to train. Training with
ResNet-50-FPN on COCO trainval35k takes 32 hours
in our synchronized 8-GPU implementation (0.72s per 16image mini-batch), and 44 hours with ResNet-101-FPN. In
fact, fast prototyping can be completed in less than one day
when training on the train set. We hope such rapid training will remove a major hurdle in this area and encourage
more people to perform research on this challenging topic.
5. Mask R-CNN for Human Pose Estimation
Our framework can easily be extended to human pose
estimation. We model a keypoint’s location as a one-hot
mask, and adopt Mask R-CNN to predict Kmasks, one for
each of Kkeypoint types ( e.g., left shoulder, right elbow).
This task helps demonstrate the ﬂexibility of Mask R-CNN.
We note that minimal domain knowledge for human pose
is exploited by our system, as the experiments are mainly to
demonstrate the generality of the Mask R-CNN framework.
We expect that domain knowledge ( e.g., modeling structures [6]) will be complementary to our simple approach.
Implementation Details: We make minor modiﬁcations to

Mask R-CNN

open source; appendix on more results


We expect that domain knowledge ( e.g., modeling structures [6]) will be complementary to our simple approach.
Implementation Details: We make minor modiﬁcations to
the segmentation system when adapting it for keypoints.
For each of the Kkeypoints of an instance, the training
target is a one-hot mmbinary mask where only a single
pixel is labeled as foreground. During training, for each visible ground-truth keypoint, we minimize the cross-entropy
loss over an m2-way softmax output (which encourages a
7
Figure 7. Keypoint detection results on COCO test using Mask R-CNN (ResNet-50-FPN), with person segmentation masks predicted
from the same model. This model has a keypoint AP of 63.1 and runs at 5 fps.
APkpAPkp
50APkp
75APkp
MAPkp
L
CMU-Pose+++ [6] 61.8 84.9 67.5 57.1 68.2
G-RMI [32]y62.4 84.0 68.5 59.1 68.1
Mask R-CNN , keypoint-only 62.7 87.0 68.4 57.4 71.1

Mask R-CNN

open source; appendix on more results


Mask R-CNN , keypoint-only 62.7 87.0 68.4 57.4 71.1
Mask R-CNN , keypoint & mask 63.1 87.3 68.7 57.8 71.4
Table 4. Keypoint detection AP on COCO test-dev . Ours is a
single model (ResNet-50-FPN) that runs at 5 fps. CMU-Pose+++
[6] is the 2016 competition winner that uses multi-scale testing,
post-processing with CPM [44], and ﬁltering with an object detector, adding a cumulative 5 points (clariﬁed in personal communication).y: G-RMI was trained on COCO plus MPII [1] (25k images), using two models (Inception-ResNet-v2 for bounding box
detection and ResNet-101 for keypoints).
single point to be detected). We note that as in instance segmentation, the Kkeypoints are still treated independently.
We adopt the ResNet-FPN variant, and the keypoint head
architecture is similar to that in Figure 4 (right). The keypoint head consists of a stack of eight 3 3 512-d conv layers, followed by a deconv layer and 2 bilinear upscaling,

Mask R-CNN

open source; appendix on more results


producing an output resolution of 56 56. We found that
a relatively high resolution output (compared to masks) is
required for keypoint-level localization accuracy.
Models are trained on all COCO trainval35k images that contain annotated keypoints. To reduce overﬁtting, as this training set is smaller, we train using image
scales randomly sampled from [640, 800] pixels; inference
is on a single scale of 800 pixels. We train for 90k iterations,
starting from a learning rate of 0.02 and reducing it by 10 at
60k and 80k iterations. We use bounding-box NMS with a
threshold of 0.5. Other details are identical as in x3.1.
Main Results and Ablations: We evaluate the person keypoint AP (APkp) and experiment with a ResNet-50-FPN
backbone; more backbones will be studied in the appendix.
Table 4 shows that our result (62.7 APkp) is 0.9 points higher
than the COCO 2016 keypoint detection winner [6] that
uses a multi-stage processing pipeline (see caption of Table 4). Our method is considerably simpler and faster.
More importantly, we have a uniﬁed model that can si-APbb
person APmask
person APkp

Mask R-CNN

open source; appendix on more results