Object Detection in Video with
Spatiotemporal Sampling Networks
Gedas Bertasius1, Lorenzo Torresani2, and Jianbo Shi1
1University of Pennsylvania,2Dartmouth College
Abstract. We propose a Spatiotemporal Sampling Network (STSN)
that uses deformable convolutions across time for object detection in
videos. Our STSN performs object detection in a video frame by learning to spatially sample features from the adjacent frames. This naturally
renders the approach robust to occlusion or motion blur in individual
frames. Our framework does not require additional supervision, as it optimizes sampling locations directly with respect to object detection performance. Our STSN outperforms the state-of-the-art on the ImageNet
VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical ﬂow data for training.
1 Introduction
In recent years, deep convolutional networks have achieved remarkable results
in many computer vision tasks [1,2,3,4,5,6,7,8], including object detection in images [9,10,11,12,13,14,15,16,17,18,19]. However, directly applying these imagelevel models to object detection in video is diﬃcult due to motion blur, video

Summary:
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.

Gedas Bertasius
Lorenzo Torresani
Jianbo Shi


defocus, unusual poses, or object occlusions (see Figure 1). Despite these challenges, it is natural to assume that video object detectors should be more powerful than still image detectors because video contains richer information about
the same object instance (e.g., its appearance in diﬀerent poses, and from different viewpoints). The key challenge then is designing a model that eﬀectively
exploits temporal information in videos.
Prior work [20,21,22,23] has proposed to exploit such temporal information
in videos by means of various post-processing steps aimed at making object detections coherent across time. However, since temporal coherence is enforced in
a second stage, typically these methods cannot be trained end-to-end. To overcome this limitation, recent work [24] has introduced a ﬂow-based aggregation
network that is trainable end-to-end. It exploits optical ﬂow to ﬁnd correspondences across time and it then aggregates features across temporal correspondences to smooth object detections over adjacent frames. However, one of the
downsides of this new model is that in addition to performing object detection,
it also needs to predict motion. This is disadvantageous due to the following reasons: 1) designing an eﬀective ﬂow network architecture is not trivial, 2) training


it also needs to predict motion. This is disadvantageous due to the following reasons: 1) designing an eﬀective ﬂow network architecture is not trivial, 2) training
such a model requires large amounts of ﬂow data, which may be diﬃcult and
costly to obtain, 3) integrating a ﬂow network and a detection network into a
single model may be challenging due to factors such as diﬀerent loss functions,
diﬀering training procedures for each network, etc. arXiv:1803.05549v2  [cs.CV]  24 Jul 2018
2 G. Bertasius, L. Torresani, J. Shi
Motion Blur
Occlusion
Unusual Pose
Video DefocusChallengingEasierChallengingEasier
Fig.1: An illustration of the common challenges associated with object detection
in video. These include video defocus, motion blur, occlusions and unusual poses.
Theboundingboxesdenotetheobjectsthatwewanttodetectintheseexamples.
To address these shortcomings, in this work, we introduce a simple, yet effective Spatiotemporal Sampling Network (STSN) that uses deformable convolutions [25] across space and time to leverage temporal information for object


detection in video. Our STSN learns to spatially sample useful feature points
from nearby video frames such that object detection accuracy in a given video
frame is maximized. To achieve this, we train our STSN end-to-end on a large set
of video frames labeled with bounding boxes. We show that this leads to a better accuracy compared to the state-of-the-art on the ImageNet VID dataset [26],
without requiring complex ﬂow network design, or the need to train the network
on large amounts of ﬂow data.
2 Related Work
2.1 Object Detection in Images
Modernobjectdetectors[9,10,11,12,13,14,15,16,17,18,19]arepredominantlybuilt
on some form of deep CNNs [1,3,5]. One of the earliest deep CNN object detection systems was R-CNN [14], which involved a two-stage pipeline where object
proposals were extracted in the ﬁrst stage, and then each proposal was classiﬁed
using a CNN. To reduce the computational burden, the methods in [9], and [13]
leveraged ROI pooling, which led to more eﬃcient learning. Furthermore, to
unify the entire object detection pipeline, Faster R-CNN [12] replaced various


leveraged ROI pooling, which led to more eﬃcient learning. Furthermore, to
unify the entire object detection pipeline, Faster R-CNN [12] replaced various
region proposal methods by another network to make the entire system trainable end-to-end. Following this work, several methods [18,19] extended Faster
R-CNN into a system that runs in real time with small reduction in performance.
Additionally, recent work [17] introduced position sensitive ROI pooling, which
signiﬁcantly improved the detection eﬃciency compared to prior object detection systems. Finally, two recent methods, Mask R-CNN [10], and Deformable
CNNs [25], improved object detection results even further and they represent
the current state-of-the-art in object detection. Whereas Mask-RCNNs use an
additional branch that predicts a mask for each region of interest, Deformable
Object Detection in Video with Spatiotemporal Sampling Networks 3
CNNs employ deformable convolutions, which allow the network to condition
discriminatively its receptive ﬁeld on the input, and to also model deformations
of objects more robustly.
While the aforementioned methods work well on images, they are not designed to exploit temporal relationships in video. Instead, our Spatiotemporal


of objects more robustly.
While the aforementioned methods work well on images, they are not designed to exploit temporal relationships in video. Instead, our Spatiotemporal
Sampling Network (STSN), is speciﬁcally designed for a video object detection
task. Unlike standard Deformable CNNs [25], which use deformable convolution
in the spatial domain, our STSN learns to sample features temporally across
diﬀerent video frames, which leads to improved video object detection accuracy.
2.2 Object Detection in Videos
Up until the introduction of the ImageNet VID challenge [26], there were no
large-scalebenchmarksforvideoobjectdetection.Thus,thereareonlyfewmethods that we can compare our work to. T-CNNs [20,21] use a video object detection pipeline that involves predicting optical ﬂow ﬁrst, then propagating imagelevel predictions according to the ﬂow, and ﬁnally using a tracking algorithm to
select temporally consistent high conﬁdence detections. Seq-NMS [22] constructs
a temporal graph from overlapping bounding box detections across the adjacent
frames, and then uses dynamic programming to select bounding box sequences
with the highest overall detection score. The work of Lee et al. [23] treats a video


frames, and then uses dynamic programming to select bounding box sequences
with the highest overall detection score. The work of Lee et al. [23] treats a video
object detection task as a multi-object tracking problem. Finally, the method of
Feichtenhofer et al. [27] proposes a ConvNet architecture that solves detection
and tracking problems jointly, and then applies a Viterbi algorithm to link the
detections across time.
The approach most similar to our work is the method of Zhu et al. [24], who
proposed an end-to-end trainable network that jointly estimates optical ﬂow and
also detects objects in video. This is accomplished by using the predicted optical
ﬂow to align the features from the adjacent frames. The aggregated features are
then fed as input to the detection network.
OurmethodisbeneﬁcialoverthemethodsthatuseopticalﬂowCNNssuchas
themethodofZhuetal.[24].First,wenotethatpretrainedopticalﬂowCNNsdo
not always generalize to new datasets, which may hinder video object detection
performance. In contrast, our method has a learnable spatiotemporal sampling
module that is discriminatively trained from object detection labels, and thus, it


performance. In contrast, our method has a learnable spatiotemporal sampling
module that is discriminatively trained from object detection labels, and thus, it
does not suﬀer from this issue. Furthermore, our STSN can be trained for video
object detection in a single stage end-to-end. In comparison, methods that rely
on optical ﬂow require an additional stage to train an optical ﬂow CNN, which
renders the training procedure more cumbersome and lengthy. For example, we
notethatitwouldtakeaboutfourdaystotrainanopticalﬂowCNNofFGFA[24]
from scratch and then four additional days to train FGFA [24] for video object
detection, making it eight days of total training time. In contrast, our STSN is
trained in a single stage in only 4 days. Finally, we point out that our STSN also
yields a gain —albeit moderate— in video object detection accuracy.
4 G. Bertasius, L. Torresani, J. Shi
3 Background: Deformable Convolution
Before describing our method, we ﬁrst review some background information on
deformable convolution [25], which is one of the key components of our STSN.
Let us ﬁrst note that a standard 2D convolution is comprised of two steps: 1)


Let us ﬁrst note that a standard 2D convolution is comprised of two steps: 1)
sampling locations on a uniformly-spaced grid R, and 2) performing a weighted
summation of sampled values using weights w. For example, if we consider a
standard 2D convolution with a 33kernel, and a dilation factor of 1, the grid
Ris deﬁned asR=f(1;1);(1;0);:::; (0;1);(1;1)g. Under a standard 2D
convolution, to compute a new value at pixel location p0in the output feature
mapy, we would perform the following operation on the input feature map x:
y(p0) =X
pn2Rw(pn)x(p0+pn);(1)
Instead, in a deformable 2D convolution, the grid Ris augmented with dataconditioned oﬀsets fpnjn= 1;:::;Ng, whereN=jRj. We can then compute
a deformable convolution as:
y(p0) =X
pn2Rw(pn)x(p0+pn+pn)(2)


a deformable convolution as:
y(p0) =X
pn2Rw(pn)x(p0+pn+pn)(2)
Since the oﬀset pnis typically fractional, the operation above is implemented using bilinear interpolation. Note that the oﬀsets are obtained by applying a separate convolutional layer to the activation tensor containing the feature
mapx.Thisyieldsanoﬀsetmapthathasthesamespatialresolutionastheinput
feature map. Also, note that the oﬀsets are shared across all feature channels of
a given activation tensor. During training, the weights for the deformable convolution kernel, and the oﬀsets kernel are learned jointly by propagating gradients
through the bilinear interpolation operator. We refer the reader to the original
work that introduced deformable convolutions [25] for further details.
4 Spatiotemporal Sampling Network
Our goal is to design a network architecture that incorporates temporal information for object detection in video.
Let us denote with Itthe frame at time tin the video. Let us consider one
of the scenarios depicted in Figure 1, e.g., a setting where Itis blurry, contains
an object in an unusual pose, or perhaps an occlusion. But let us assume that


of the scenarios depicted in Figure 1, e.g., a setting where Itis blurry, contains
an object in an unusual pose, or perhaps an occlusion. But let us assume that
a nearby frame It+kincludes the same object clearly visible and in a relatively
standard pose. If we only had access to It, accurate object detection would be
very challenging. However, leveraging information from It+kmay enable more
robust detection in the frame It. Thus, the main challenge in this setting is
incorporating object-level information from the supporting frameIt+kfor an improved object detection accuracy in the reference frameIt. Note that in our
system each frame in the video is treated in turn as a reference frame in order to
produce object detection in every frame of the video. Furthermore, in practice
Object Detection in Video with Spatiotemporal Sampling Networks 5
Reference Frame (time t)
Supporting Frame (time t+k)
Def. Conv.OffsetsDef. Conv.Offsets
Sampled Feature Map
Concat
Fig.2: Our spatiotemporal sampling mechanism, which we use for video object
detection. Given the task of detecting objects in a particular video frame (i.e.,
a reference frame), our goal is to incorporate information from a nearby frame
of the same video (i.e., a supporting frame). First, we extract features from


a reference frame), our goal is to incorporate information from a nearby frame
of the same video (i.e., a supporting frame). First, we extract features from
both frames via a backbone convolutional network (CNN). Next, we concatenate
the features from the reference and supporting frames, and feed them through
multiple deformable convolutional layers. The last of such layers produces oﬀsets
that are used to sample informative features from the supporting frame. Our
spatiotemporal sampling scheme allows us to produce accurate detections even
if objects in the reference frame appear blurry or occluded.
we use 2Ksupporting frames for detection in the reference frame, by taking
theKpreceding frames and the Ksubsequent frames as supporting frames, i.e.
fItK;It(K1);:::;I t1;It+1;:::;I t+(K1);It+Kg. However, for ease of explanation we introduce our STSN by considering a single supporting frame It+k.
To eﬀectively integrate temporal information we need two things: 1) powerful
object-level features from an image-level network, and 2) an ability to sample
useful object-level features from the supporting frames for the reference frame.


object-level features from an image-level network, and 2) an ability to sample
useful object-level features from the supporting frames for the reference frame.
Weachievetheformerbyemployingastate-of-the-artbackbonenetwork.Forthe
latter, we design a spatiotemporal sampling scheme, which we describe below.
Our STSN can be summarized in four steps. First, a backbone convolutional
network computes object-level features for each video frame individually. Then,
spatiotemporal sampling blocks are applied to the object-level feature maps in
order to sample relevant features from nearby frames conditioned on the input
reference frame.Next,thesampledfeaturesfromeachvideoframearetemporally
aggregated into a single feature tensor for the reference frame using a per-pixel
weighted summation. Finally, the feature tensor is provided as input to the
detectionnetworktoproduceﬁnalobjectdetectionresultsforthegivenreference
frame. We note that our framework integrates these conceptually-distinct four
steps into a single architecture, which we train end-to-end.
6 G. Bertasius, L. Torresani, J. Shi
Backbone Architecture. Our backbone network is applied to each frame


6 G. Bertasius, L. Torresani, J. Shi
Backbone Architecture. Our backbone network is applied to each frame
of the video. As backbone network, we use a Deformable CNN [25] based on
the ResNet-101 [5] architecture, which is one of the top-performing object detection systems at the moment. Similarly to [25], our backbone network employs
6deformable convolutional layers. We also note that even though we use a Deformable CNN architecture, our system can easily integrate other architectures
and thus it can beneﬁt from future improvements in still-image object detection.
Spatiotemporal Feature Sampling. Our main contribution is the design
of a spatiotemporal sampling mechanism, which seamlessly integrates temporal
information in a given video. As a ﬁrst step, we feed the reference frame Itand
the supporting frame It+kthrough our image-level backbone network, which
produces feature tensors ftandft+k, respectively. Note that ft;ft+k2Rchw
wherec;h, andware the number of channels, the height, and the width of the
activation tensor. The feature tensors ft, andft+kare then concatenated into a


activation tensor. The feature tensors ft, andft+kare then concatenated into a
new feature tensor ft;t+k2R2chw. Note that this tensor ft;t+know has twice
as many channels as our initial tensors, and that it now contains object-level
information from both the reference and the supporting frame.
Next, we use the tensor ft;t+kto predict (x;y)location oﬀsets, which are
then used to sample the supporting tensor ft+k. The sampling mechanism is
implementedusingadeformableconvolutionallayer,whichtakes1)thepredicted
oﬀsets, and 2) the supporting tensor ft+kas its inputs, and then outputs a
newly sampled feature tensor gt;t+k, which can be used for object detection in
the reference frame. We use subscript t;t+kto denote the resampled tensor
because, although gis obtained by resampling the supporting tensor, the oﬀset
computation uses both the reference as well as the supporting frame. A detailed
illustration of our spatiotemporal sampling scheme is presented in Figure 2.
In practice, our spatiotemporal sampling block has 4deformable convolution


illustration of our spatiotemporal sampling scheme is presented in Figure 2.
In practice, our spatiotemporal sampling block has 4deformable convolution
layers (only 2are shown in Figure 2). This means that the initially predicted
oﬀsetso(1)
t;t+kand the concatenated temporal features ft;t+kare ﬁrst used as
inputs to a deformable convolution layer that outputs a new feature map g(1)
t;t+k.
Next, we use g(1)
t;t+kto predict oﬀsets o(2)
t;t+k, and a new feature map g(2)
t;t+k. This
continues for 2 more layers until we obtain oﬀsets o(4)
t;t+k, which are then used
to sample the points out of the supporting feature map ft+k. The ﬁnal sampled
feature map g(4)
t;t+kis obtained via another deformable convolutional layer that
takes as inputs oﬀsets o(4)
t;t+kand the original supporting feature map ft+k.
Our proposed spatiotemporal sampling mechanism learns, which object-level
features in the supporting frame are useful for object detection in the reference


Our proposed spatiotemporal sampling mechanism learns, which object-level
features in the supporting frame are useful for object detection in the reference
frame. Conceptually, it replaces the optical ﬂow used in [24] to establish temporal correspondences with a learnable module that is discriminatively trained
from object detection labels. In our experimental section, we show that such a
sampling scheme allows us to improve video object detection performance over
the still-image baseline and the ﬂow-based method of Zhu et al. [24] without
training our model on optical ﬂow data.
Feature Aggregation. The spatiotemporal sampling procedure is applied
for all the supporting frames in the selected range. Note, that this includes a
Object Detection in Video with Spatiotemporal Sampling Networks 7
special case, when the reference frame is treated as a supporting frame to itself to
produceg(4)
t;t, which is a feature tensor computed from only the reference frame.
The resulting feature tensors have the following form: g(4)
t;t+k2Rc(4)hw.
Thesefeaturetensorsareaggregatedintoanoutputfeaturetensor gagg
t2Rc(4)hw
for the reference frame. This tensor captures information from the reference


t2Rc(4)hw
for the reference frame. This tensor captures information from the reference
frame, itsKpreceding frames and its Ksubsequent frames. The output tensor
valuegagg
t(p)for frametat pixelpis computed as a weighted summation:
gagg
t(p) =KX
k=Kwt;t+k(p)g(4)
t;t+k(p) (3)
Inspired by strong results presented in [24], we use their proposed feature
aggregation method where the weights windicate the importance of each supporting frame to the reference frame. To compute the weights w, we attach a
3-layer subnetwork S(x)to the features g(4)
t;t+kand then compute their intermediate feature representations S(g(4)
t;t+k). We then obtain the weights wby applying
an exponential function on the cosine similarity between each corresponding
feature point in a reference frame and a supporting frame:
wt;t+k(p) = exp
S(g(4)
t;t)(p)S(g(4)
t;t+k)(p)
jS(g(4)
t;t)(p)jjS(g(4)


t;t+k)(p)
jS(g(4)
t;t)(p)jjS(g(4)
t;t+k)(p)j!
(4)
Finally,allweights warefedintothesoftmaxlayer,toensurethattheweights
sum up to 1at each pixel location p(i.e.,PK
k=Kwt;t+k(p) = 18p).
Object Detection. Finally, the aggregated feature tensor gagg
tis used as
input to the detection network, which outputs the ﬁnal bounding box predictions
and their object class probabilities. We describe more details related to the
detection network in the next section along with other implementation details.
4.1 Implementation Details
For our experiments we use the MXNet [28] library. Below we provide details
related to our STSN architecture, and our training and inference procedures.
Architecture. For our backbone network we adopt a state-of-the-art Deformable CNN [25] based on the ResNet-101 [5] architecture. Our spatiotemporal sampling block consists of four 33deformable convolutional layers each
with 1024output channels. In addition, it also has four 33convolutional layers


with 1024output channels. In addition, it also has four 33convolutional layers
predicting (x;y)oﬀsets. To implement a subnetwork S(x)that predicts feature
aggregation weights, we use a sequence of 11,33and11convolutional
layers with 512;512and2048output channels respectively. Our detection network is implemented based on the deformable R-FCN design [17,29,25]. When
feeding the aggregated feature gagg
tto the detection network, we split its 1024
channels into two parts, and feed the ﬁrst and the last 512channels to the
RPN and R-FCN sub-networks respectively. For the RPN, we use 9anchors and
300proposals for each image. Furthermore, for the R-FCN, we use deformable
position-sensitive ROI pooling with 77groups.
8 G. Bertasius, L. Torresani, J. Shi
Training. Our entire STSN model is fully diﬀerentiable, and thus, trainable
end-to-end. During training, we resize all input images to a shorter side of 600
pixels, and use T= 3frames to train our model (i.e., K= 1). More speciﬁcally,


pixels, and use T= 3frames to train our model (i.e., K= 1). More speciﬁcally,
we randomly sample one supporting frame before and one supporting frame after
the reference frame. We observed that using more supporting frames in training
does not lead to a higher accuracy.
For the rest of our training procedure, we follow the protocol outlined in [24].
Speciﬁcally, we train our model in two stages. First, we pre-train our full model
on the Imagenet DET dataset using the annotations of the 30object classes
that overlap with the Imagenet VID dataset. Note that Imagenet DET dataset
contains only images, and thus, we cannot sample meaningful supporting frames
in this case. Therefore, in the case of images, we use the reference frames as our
supportingframes.Afterwards,theentiremodelistrainedfor 120Kiterationson
4Tesla K40 GPUs with each GPU holding a single mini-batch. The learning rate
is set to 0:001and0:0001for the ﬁrst 80Kand the last 40Kiterations respectively. Afterwards, we ﬁnetune the entire model on the Imagenet VID dataset
for60Kiterations with a learning rate of 0:001and0:0001for the ﬁrst 40Kand


for60Kiterations with a learning rate of 0:001and0:0001for the ﬁrst 40Kand
the last 20Kiterations respectively. Note that in the second stage of training
we sample the supporting frames randomly within a certain neighborhood of a
reference frame (as described above).
Inference. During inference, we use T= 27, meaning that we consider
K= 13supporting frames before and after the reference frame. To avoid GPU
memory issues, we ﬁrst extract features from the backbone network for each
image individually, and then cache these features in the memory. Afterwards,
we feed all these features into our spatiotemporal sampling block. At the end,
standard NMS with a threshold of 0:3is applied to reﬁne the detections. To
handle the ﬁrst and the last K= 13frames in the video —two boundary cases
that require sampling the neighboring frames beyond the video start and end,
we pad the start of a video with Kcopies of the ﬁrst frame, and the end of a
video with Kcopies of the last frame.
5 Experimental Results
In this section, we evaluate our approach for video object detection on the ImageNet VID [26] dataset, which has 3;862and555training and testing video


5 Experimental Results
In this section, we evaluate our approach for video object detection on the ImageNet VID [26] dataset, which has 3;862and555training and testing video
clips respectively. Each video is annotated with bounding boxes. The frames
from each video are extracted at 2530fps. The dataset contains 30object
categories that are a subset of the 200categories in the ImageNet DET dataset.
5.1 Quantitative Results
To assess the eﬀectiveness of our method we compare it to several relevant baselines, mainly two state-of-the-art methods FGFA [24] and D&T [27]. First, to
verify that using temporal information from video is beneﬁcial, we include a
static image-level variant of our model (SSN) that uses only the reference frame
Object Detection in Video with Spatiotemporal Sampling Networks 9
Methods
D&T [27] Our SSN FGFA [24] Our STSN D&T+ [27] Our STSN+
No FlowNet? 3 - 7 3 3 3
Not Using Flow Data? 3 - 7 3 3 3
No Temporal Post-Processing? 3 - 3 3 7 7
mAP@0.5 75.8 76.0 78.8 78.9 79.8 80.4


No Temporal Post-Processing? 3 - 3 3 7 7
mAP@0.5 75.8 76.0 78.8 78.9 79.8 80.4
Table 1: We use the ImageNet VID [26] dataset to compare our STSN to the
state-of-the-art FGFA [24] and D&T [27] methods. Note that SSN refers to our
static baseline, which is obtained by using only the reference frame for output
generation (no temporal info). Also note, that D&T+ and STSN+ refer to D&T
and STSN baselines with temporal post-processing applied on top of the CNN
outputs. Based on these results, we ﬁrst point out that unlike FGFA, our STSN
does not rely on the external optical ﬂow data, and still yields higher mAP
(78:9vs78:8). Furthermore, when no temporal post-processing is used, our
STSN produces superior performance in comparison to the D&T baseline ( 78:9
vs75:8). Finally, we demonstrate that if we use a simple Seq-NMS [22] temporal
post-processing scheme on top of our STSN predictions, we can further improve
our results and outperform all the other baselines.


post-processing scheme on top of our STSN predictions, we can further improve
our results and outperform all the other baselines.
to make its predictions. Furthermore, we also want to show that our spatiotemporal sampling scheme works as eﬀectively as the optical ﬂow network in [24],
but without requiring optical ﬂow supervision. To do so, we replace the optical
ﬂow network from [24], with our spatiotemporal sampling mechanism. The rest
of the architecture and the training details are kept the same for both baselines.
Such an experimental design allows us to directly compare the eﬀectiveness of
our spatiotemporal sampling scheme and the optical ﬂow network of FGFA [24].
Finally, we demonstrate that our method performs better than the D&T [27]
method in two scenarios: 1) when we only useCNN-level outputs forvideo object
detection, and also 2) when we allow temporal post-processing techniques such
as Seq-NMS to be applied on top of the CNN outputs. We note that in Table 1,
D&T [27] and STSN refer to the CNN-level baselines whereas D&T+ [27] and


D&T [27] and STSN refer to the CNN-level baselines whereas D&T+ [27] and
STSN+ denote these same methods but with temporal post-processing (i.e. SeqNMS[22],object-tubebasedlinking[27],etc)appliedontopoftheCNNoutputs.
We present our results in Table 1, where we assess each method according to
several criteria. In the ﬁrst row of Table 1, we list whether a given method requires integrating a separate ﬂow network into its training / prediction pipeline.
Ideally, we would want to eliminate this step because optical ﬂow prediction requiresdesigningahighlycomplexﬂownetworkarchitecture.Wealsolistwhether
a given method requires pre-training on the external optical ﬂow data, which we
would want to avoid since it makes the whole training pipeline more costly.
Additionally, we list, whether a given method uses any external temporal postprocessing steps, which we would want to eliminate because they typically make
the training / prediction pipeline disjoint and more complex. Finally, we assess
each method according to the standard mean average precision (mAP) metric
at intersection over union (IoU) threshold of 0:5.
10 G. Bertasius, L. Torresani, J. Shi


at intersection over union (IoU) threshold of 0:5.
10 G. Bertasius, L. Torresani, J. Shi
0 2 6 10 14 18 22 26 30Number of Supporting Frames7676.57777.57878.579mAP
-10 -8 -6 -4 -2 0 2 4 6 8 10Time Delta w.r.t Reference Frame0.060.0650.070.0750.080.0850.090.0950.1Weight Magnitude
Fig.3: A ﬁgure illustrating some of our ablation experiments. Left:we plot mAP
as a function of the number of supporting frames used by our STSN. From this
plot, we notice that the video object detection accuracy improves as we use
more supporting frames. Right:To understand the contribution of each of the
supporting frames, we plot the average weight magnitudes wt;t+k(p)for diﬀerent
values ofk. Here,prepresents a point at the center of an object. From this plot,
we observe that the largest weights are associated with the supporting frames
that are near the reference frame. However, note that even supporting frames
that are further away from the reference frame (e.g. k= 9) contribute quite
substantially to the ﬁnal object detection predictions.


that are further away from the reference frame (e.g. k= 9) contribute quite
substantially to the ﬁnal object detection predictions.
Based on our results in Table 1, we make the following conclusions. First, we
notethatourSTSNproducesbetterquantitativeresultsthanthestate-of-the-art
FGFA method ( 78:9vs78:8). We acknowledge that our accuracy improvement
over FGFA is moderate. However, we point out that our STSN operates in a
much more challenging setting than FGFA. Unlike FGFA, our STSN does not
use any optical ﬂow supervision. Instead, it is trained directly for video object
detection. The fact that STSN learns temporal correspondences without direct
optical ﬂow supervision, and still outperforms FGFA is quite impressive. Such
results also show the beneﬁt of discriminative end-to-end training with respect
to the ﬁnal video object detection task objective.
WenextcompareourSTSNtotheD&Tbaseline[27].Wenotethatunlikefor
the FGFA [24] baseline, it is much harder to make a direct comparison between


the FGFA [24] baseline, it is much harder to make a direct comparison between
STSN and D&T. Whereas our STSN aims to produce powerful spatiotemporal features, the method of D&T [27] is targeted more for smoothing the ﬁnal
bounding box predictions across time. Thus, we believe that these two methods
are complementary, and it would be possible to integrate them together for the
model that produces both: temporally smooth features, as well as temporally
smooth bounding box predictions. We also note that our STSN and D&T [27]
use slightly diﬀerent architectures (both based on ResNet-101 though).
First, we compare STSN and D&T in a setting when no temporal postprocessing(i.e.Seq-NMS[22],object-tubelinking[27],etc)isused,andshowthat
ourSTSNoutperformstheD&Tbaselinebyasubstantialmargin( 78:9vs75:8).
Object Detection in Video with Spatiotemporal Sampling Networks 11
These results indicate, that our STSN is able to learn powerful spatiotemporal
features, and produce solid video object detection results even without temporal
post-processing algorithms that link bounding box detections over time.


features, and produce solid video object detection results even without temporal
post-processing algorithms that link bounding box detections over time.
Afterwards, we show that integrating a simple temporal post-processing algorithm Seq-NMS [22] further improves our STSN’s results. Such a scheme allows
us to outperform the D&T+ baseline ( 80:4vs79:8), which uses a similar Viterbi
based temporal post-processing scheme.
5.2 Ablation Studies
Optimal Number of Supporting Frames. In the left subplot of Figure 3,
we also illustrate how the number of supporting frames aﬀects the video object
detection accuracy. We notice that the performance keeps increasing as we add
more supporting frames, and then plateaus at T= 27.
Increasing the Temporal Stride. We also investigate how the temporal
stridek, at which we sample the supporting frames, aﬀects STSN’s performance.
We report that temporal strides of k= 2andk= 4, yield mAP scores of 79:0
and77:9, respectively. Thus, k= 2yields a slight improvement over our original
78:9mAP score. However, increasing kto larger values reduces the accuracy.
Feature Aggregation Weight Analysis. To analyze how much each of


78:9mAP score. However, increasing kto larger values reduces the accuracy.
Feature Aggregation Weight Analysis. To analyze how much each of
the supporting frame contributes to the ﬁnal object detections, we visualize the
average weight magnitudes wt;t+k(p)for diﬀerent values of k. This visualization
is presented in the right subplot of Figure 3. We note that in this case, the weight
magnitudes correspond to the point p, which is located at the center of an object.
From this plot, we can conclude that the largest contribution comes from the
supporting frames that are near the reference frame ( k=1;0;1). However,
note that even even supporting frames that are further away from the reference
frame (e.g. k=9;9) have non-zero weights, and contribute quite substantially
to the ﬁnal object detection predictions.
5.3 Qualitative Results
To understand how our STSN exploits temporal information from a given video,
we visualize in Figure 4, the average oﬀsets predicted by the STSN sampling
block. These oﬀsets are used by the STSN to decide, which object-level informationfromthesupportingframeshouldbeusedtodetectanobjectinthereference


frame. The green square in the reference frame depicts a pixel, for which we want
to compute a convolution output. The red square in the supporting frame represents an average oﬀset, which is used to determine which feature points from the
supporting frame should be sampled. The yellow arrow indicates object’s motion between the reference frame and the supporting frame. Note that despite
a relatively large motion between the reference and the supporting frames, our
STSN samples features from the supporting frame right around the center of the
object, which is exactly what we want. Such spatiotemporal sampling allows us
to detect objects even if they appear blurry or occluded in the reference frame.
12 G. Bertasius, L. Torresani, J. Shi
Reference Frame (t)Supporting Frame (t+9)
Reference Frame (t)Supporting Frame (t+9)
Fig.4: An illustration of our spatiotemporal sampling scheme (zoom-in for a better view). The green square indicates a point in the reference frame, for which
we want to compute a new convolutional output. The red square indicates the
corresponding point predicted by our STSN in a supporting frame. The yellow
arrow illustrates the estimated object motion. Although our model is trained
discriminatively for object detection and notfor tracking or motion estimation,


arrow illustrates the estimated object motion. Although our model is trained
discriminatively for object detection and notfor tracking or motion estimation,
our STSN learns to sample from the supporting frame at locations that coincide almost perfectly with the same object. This allows our method to perform
accurate object detection even if objects in the reference frame are blurry or
occluded.
In addition, based on the results in Figure 4, we observe that even without
an explicit optical ﬂow supervision, our STSN learns to accurately capture the
motion of the objects, which is another appealing property of our model. In
fact, in Figure 5, we illustrate several examples of using our STSN to track
objects in a given video. From Figure 5, we observe that despite a relatively
large motion in each sequence, our STSN accurately samples features around
objects in every supporting frame. Such results indicate that we may be able
to use our sampling mechanism for discriminative object tracking. In fact, we
note that the commonly used dense optical ﬂow methods are often redundant
because most applications do not require ﬂow prediction for every single pixel. In
comparison, we point out that our STSN captures a more discriminative form of
motion, which is learned to exclusively beneﬁt a video object detection task. In
our supplementary material, we include more of such results in the video form.


motion, which is learned to exclusively beneﬁt a video object detection task. In
our supplementary material, we include more of such results in the video form.
In Figure 6, we also illustrate object detections of the static SSN baseline,
and those of our full STSN model (zoom-in to see the probabilities and class
predictions). In all of these cases, we observe that incorporating temporal information helps STSN to correct the mistakes made by the static baseline. For
instance, in the third row of Figure 6, a static SSN baseline incorrectly labels
an object in the reference frame as a bird, which happens due to the occluded
head of the lizard. However, STSN ﬁxes this mistake by looking at the supporting frames, and by sampling around the lizard body and its head (See Row 3,
Column 1 in Figure 6). Furthermore, in the last row, a static SSN baseline fails
to detect one of the bicycles because it is occluded in the reference frame. STSN
ﬁxes this error, by sampling around the missed bicycle in the supporting frame
Object Detection in Video with Spatiotemporal Sampling Networks 13
Reference Frame (t)Supporting Frame (t-9)Supporting Frame (t-4)Supporting Frame (t+4)Supporting Frame (t+9)


Reference Frame (t)Supporting Frame (t-9)Supporting Frame (t-4)Supporting Frame (t+4)Supporting Frame (t+9)
Fig.5: An illustration of using our spatiotemporal sampling scheme in action.
The green square indicates a ﬁxed object location in the reference frame. The
red square depicts a location in a supporting frame, from which relevant features
are sampled. Even without optical ﬂow supervision, our STSN learns to track
these objects in video. In our supplementary material, we include more of such
examples in the video format.
where the bicycle is more clearly visible. Similar behavior also occurs in other
cases where STSN successfully resolves occlusion and blurriness issues.
6 Conclusion
Inthiswork,weintroducedtheSpatiotemporalSamplingNetwork(STSN)which
is a new architecture for object detection in video. Compared to the state-of-theart FGFA [24] method, our model involves a simpler design, it does not require
optical ﬂow computation and it produces higher video object detection accuracy.
Ourmodelisfullydiﬀerentiable,andunlikepriorvideoobjectdetectionmethods,
it does not necessitate optical ﬂow training data. This renders our model easy to


it does not necessitate optical ﬂow training data. This renders our model easy to
train end-to-end. Our future work will include experimenting with more complex
design of spatiotemporal sampling blocks.
7 Acknowledgements
This work was funded in part by NSF award CNS-120552. We gratefully acknowledge NVIDIA and Facebook for the donation of GPUs used for portions
of this work.
14 G. Bertasius, L. Torresani, J. Shi
SSN (Static)STSN (Ours)
Sampled Points in the     Supporting Frame
Fig.6: A ﬁgure illustrating object detection examples where our spatiotemporal
sampling mechanism helps STSN to correct the mistakes made by a static SSN
baseline (please zoom-in to see the class predictions and their probabilities).
These mistakes typically occur due to occlusions, blurriness, etc. STSN ﬁxes
these errors by using relevant object level information from supporting frames.
InColumn 1weillustratethepointsinthesupportingframethatSTSNconsiders
relevant when computing the output for a point denoted by the green square in
Column 2.
Object Detection in Video with Spatiotemporal Sampling Networks 15
References


relevant when computing the output for a point denoted by the green square in
Column 2.
Object Detection in Video with Spatiotemporal Sampling Networks 15
References
1. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classiﬁcation with deep
convolutional neural networks. In: NIPS. (2012)
2. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D.,
Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Computer
Vision and Pattern Recognition (CVPR). (2015)
3. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale
image recognition. In: ICLR. (2015)
4. Bertasius, G., Shi, J., Torresani, L.: Deepedge: A multi-scale bifurcated deep
network for top-down contour detection. In: The IEEE Conference on Computer
Vision and Pattern Recognition (CVPR). (June 2015)


network for top-down contour detection. In: The IEEE Conference on Computer
Vision and Pattern Recognition (CVPR). (June 2015)
5. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. 2016IEEEConferenceonComputerVisionandPatternRecognition(CVPR)
(2016) 770–778
6. Xie,S.,Girshick,R.,Dollár,P.,Tu,Z.,He,K.: Aggregatedresidualtransformations
for deep neural networks. In: CVPR. (2017)
7. Toshev, A., Szegedy, C.: Deeppose: Human pose estimation via deep neural networks. In: CVPR. (2014)
8. Bertasius, G., Torresani, L., Yu, S.X., Shi, J.: Convolutional random walk networks
for semantic image segmentation. In: The IEEE Conference on Computer Vision
and Pattern Recognition (CVPR). (July 2017)
9. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional
networks for visual recognition. In Fleet, D., Pajdla, T., Schiele, B., Tuytelaars,


networks for visual recognition. In Fleet, D., Pajdla, T., Schiele, B., Tuytelaars,
T., eds.: Computer Vision – ECCV 2014. (2014)
10. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of
the International Conference on Computer Vision (ICCV). (2017)
11. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal Loss for Dense Object
Detection. In: Proceedings of the International Conference on Computer Vision
(ICCV). (2017)
12. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Neural Information Processing Systems
(NIPS). (2015)
13. Girshick, R.: Fast R-CNN. In: Proceedings of the International Conference on
Computer Vision (ICCV). (2015)
14. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR). (2014)


Conference on Computer Vision and Pattern Recognition (CVPR). (2014)
15. Gupta,S.,Girshick,R.,Arbelaez,P.,Malik,J.: LearningrichfeaturesfromRGB-D
images for object detection and segmentation. In: ECCV. (2014)
16. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.:
Ssd: Single shot multibox detector. In: ECCV. (2016)
17. Dai, J., Li, Y., He, K., Sun, J.: R-fcn: Object detection via region-based fully
convolutional networks. In: Advances in Neural Information Processing Systems
29. Curran Associates, Inc. (2016) 379–387
18. Redmon,J.,Divvala,S.K.,Girshick,R.B.,Farhadi,A.: Youonlylookonce:Uniﬁed,
real-time object detection. In: 2016 IEEE Conference on Computer Vision and
Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. (2016)
779–788
16 G. Bertasius, L. Torresani, J. Shi


779–788
16 G. Bertasius, L. Torresani, J. Shi
19. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: 2017 IEEE
Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu,
HI, USA, July 21-26, 2017. (2017) 6517–6525
20. Kang, K., Li, H., Yan, J., Zeng, X., Yang, B., Xiao, T., Zhang, C., Wang, Z., Wang,
R., Wang, X., Ouyang, W.: T-CNN: tubelets with convolutional neural networks
for object detection from videos. IEEE TCSVT 2017 (2017)
21. Kang, K., Ouyang, W., Li, H., Wang, X.: Object detection from video tubelets
with convolutional neural networks. CoRR abs/1604.04053 (2016)
22. Han, W., Khorrami, P., Paine, T.L., Ramachandran, P., Babaeizadeh, M., Shi,
H., Li, J., Yan, S., Huang, T.S.: Seq-nms for video object detection. CoRR
abs/1602.08465 (2016)


H., Li, J., Yan, S., Huang, T.S.: Seq-nms for video object detection. CoRR
abs/1602.08465 (2016)
23. Lee, B., Erdenee, E., Jin, S., Rhee, P.: Multi-class multi-object tracking using
changing point detection. CoRR abs/1608.08434 (2016)
24. Zhu, X., Wang, Y., Dai, J., Yuan, L., Wei, Y.: Flow-guided feature aggregation for
video object detection. In: International Conference on Computer Vision (ICCV).
(2017)
25. Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y.: Deformable convolutional networks. In: 2017 IEEE International Conference on Computer Vision
(ICCV). Volume 00. (Oct. 2017) 764–773
26. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,
Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large
Scale Visual Recognition Challenge. International Journal of Computer Vision
(IJCV) 115(3) (2015) 211–252


Scale Visual Recognition Challenge. International Journal of Computer Vision
(IJCV) 115(3) (2015) 211–252
27. Feichtenhofer, C., Pinz, A., Zisserman, A.: Detect to track and track to detect. In:
International Conference on Computer Vision (ICCV). (2017)
28. Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang,
C., Zhang, Z.: Mxnet: A ﬂexible and eﬃcient machine learning library for heterogeneous distributed systems. CoRR abs/1512.01274 (2015)
29. Zhu, X., Xiong, Y., Dai, J., Yuan, L., Wei, Y.: Deep feature ﬂow for video recognition. In: CVPR. (2017)


Relational inductive biases, deep learning, and graph networks
Peter W. Battaglia1,Jessica B. Hamrick1,Victor Bapst1,
Alvaro Sanchez-Gonzalez1,Vinicius Zambaldi1,Mateusz Malinowski1,
Andrea Tacchetti1,David Raposo1,Adam Santoro1,Ryan Faulkner1,
Caglar Gulcehre1,Francis Song1,Andrew Ballard1,Justin Gilmer2,
George Dahl2,Ashish Vaswani2,Kelsey Allen3,Charles Nash4,
Victoria Langston1,Chris Dyer1,Nicolas Heess1,
Daan Wierstra1,Pushmeet Kohli1,Matt Botvinick1,
Oriol Vinyals1,Yujia Li1,Razvan Pascanu1
1DeepMind;2Google Brain;3MIT;4University of Edinburgh
Abstract
Articial intelligence (AI) has undergone a renaissance recently, making major progress in
key domains such as vision, language, control, and decision-making. This has been due, in
part, to cheap data and cheap compute resources, which have t the natural strengths of deep
learning. However, many dening characteristics of human intelligence, which developed under

Summary:
Artificial intelligence (AI) has undergone a renaissance recently, making
major progress in key domains such as vision, language, control, and
decision-making. This has been due, in part, to cheap data and cheap compute
resources, which have fit the natural strengths of deep learning. However, many
defining characteristics of human intelligence, which developed under much
different pressures, remain out of reach for current approaches. In particular,
generalizing beyond one's experiences--a hallmark of human intelligence from
infancy--remains a formidable challenge for modern AI.
  The following is part position paper, part review, and part unification. We
argue that combinatorial generalization must be a top priority for AI to
achieve human-like abilities, and that structured representations and
computations are key to realizing this objective. Just as biology uses nature
and nurture cooperatively, we reject the false choice between
"hand-engineering" and "end-to-end" learning, and instead advocate for an
approach which benefits from their complementary strengths. We explore how
using relational inductive biases within deep learning architectures can
facilitate learning about entities, relations, and rules for composing them. We
present a new building block for the AI toolkit with a strong relational
inductive bias--the graph network--which generalizes and extends various
approaches for neural networks that operate on graphs, and provides a
straightforward interface for manipulating structured knowledge and producing
structured behaviors. We discuss how graph networks can support relational
reasoning and combinatorial generalization, laying the foundation for more
sophisticated, interpretable, and flexible patterns of reasoning. As a
companion to this paper, we have released an open-source software library for
building graph networks, with demonstrations of how to use them in practice.

Peter W. Battaglia
Jessica B. Hamrick
Victor Bapst
Alvaro Sanchez-Gonzalez
Vinicius Zambaldi
Mateusz Malinowski
Andrea Tacchetti
David Raposo
Adam Santoro
Ryan Faulkner
Caglar Gulcehre
Francis Song
Andrew Ballard
Justin Gilmer
George Dahl
Ashish Vaswani
Kelsey Allen
Charles Nash
Victoria Langston
Chris Dyer
Nicolas Heess
Daan Wierstra
Pushmeet Kohli
Matt Botvinick
Oriol Vinyals
Yujia Li
Razvan Pascanu


part, to cheap data and cheap compute resources, which have t the natural strengths of deep
learning. However, many dening characteristics of human intelligence, which developed under
much dierent pressures, remain out of reach for current approaches. In particular, generalizing
beyond one's experiences|a hallmark of human intelligence from infancy|remains a formidable
challenge for modern AI.
The following is part position paper, part review, and part unication. We argue that
combinatorial generalization must be a top priority for AI to achieve human-like abilities, and that
structured representations and computations are key to realizing this objective. Just as biology
uses nature and nurture cooperatively, we reject the false choice between \hand-engineering"
and \end-to-end" learning, and instead advocate for an approach which benets from their
complementary strengths. We explore how using relational inductive biases within deep learning
architectures can facilitate learning about entities, relations, and rules for composing them. We
present a new building block for the AI toolkit with a strong relational inductive bias|the graph
network |which generalizes and extends various approaches for neural networks that operate
on graphs, and provides a straightforward interface for manipulating structured knowledge and
producing structured behaviors. We discuss how graph networks can support relational reasoning


on graphs, and provides a straightforward interface for manipulating structured knowledge and
producing structured behaviors. We discuss how graph networks can support relational reasoning
and combinatorial generalization, laying the foundation for more sophisticated, interpretable,
and
exible patterns of reasoning. As a companion to this paper, we have also released an
open-source software library for building graph networks, with demonstrations of how to use
them in practice.
1 Introduction
A key signature of human intelligence is the ability to make \innite use of nite means" (Humboldt,
1836; Chomsky, 1965), in which a small set of elements (such as words) can be productively
composed in limitless ways (such as into new sentences). This re
ects the principle of combinatorial
generalization , that is, constructing new inferences, predictions, and behaviors from known building
blocks. Here we explore how to improve modern AI's capacity for combinatorial generalization by
Corresponding author: peterbattaglia@google.com
1arXiv:1806.01261v3  [cs.LG]  17 Oct 2018
biasing learning towards structured representations and computations, and in particular, systems
that operate on graphs.


biasing learning towards structured representations and computations, and in particular, systems
that operate on graphs.
Humans' capacity for combinatorial generalization depends critically on our cognitive mechanisms for representing structure and reasoning about relations. We represent complex systems as
compositions of entities and their interactions1(Navon, 1977; McClelland and Rumelhart, 1981;
Plaut et al., 1996; Marcus, 2001; Goodwin and Johnson-Laird, 2005; Kemp and Tenenbaum, 2008),
such as judging whether a haphazard stack of objects is stable (Battaglia et al., 2013). We use
hierarchies to abstract away from ne-grained dierences, and capture more general commonalities
between representations and behaviors (Botvinick, 2008; Tenenbaum et al., 2011), such as parts of
an object, objects in a scene, neighborhoods in a town, and towns in a country. We solve novel
problems by composing familiar skills and routines (Anderson, 1982), for example traveling to a
new location by composing familiar procedures and objectives, such as \travel by airplane", \to
San Diego", \eat at", and \an Indian restaurant". We draw analogies by aligning the relational
structure between two domains and drawing inferences about one based on corresponding knowledge


San Diego", \eat at", and \an Indian restaurant". We draw analogies by aligning the relational
structure between two domains and drawing inferences about one based on corresponding knowledge
about the other (Gentner and Markman, 1997; Hummel and Holyoak, 2003).
Kenneth Craik's \The Nature of Explanation" (1943), connects the compositional structure of
the world to how our internal mental models are organized:
...[a human mental model] has a similar relation-structure to that of the process it
imitates. By `relation-structure' I do not mean some obscure non-physical entity which
attends the model, but the fact that it is a working physical model which works in the
same way as the process it parallels... physical reality is built up, apparently, from a few
fundamental types of units whose properties determine many of the properties of the
most complicated phenomena, and this seems to aord a sucient explanation of the
emergence of analogies between mechanisms and similarities of relation-structure among
these combinations without the necessity of any theory of objective universals. (Craik,
1943, page 51-55)
That is, the world is compositional, or at least, we understand it in compositional terms. When
learning, we either t new knowledge into our existing structured representations, or adjust the


learning, we either t new knowledge into our existing structured representations, or adjust the
structure itself to better accommodate (and make use of) the new and the old (Tenenbaum et al.,
2006; Griths et al., 2010; Ullman et al., 2017).
The question of how to build articial systems which exhibit combinatorial generalization has
been at the heart of AI since its origins, and was central to many structured approaches, including
logic, grammars, classic planning, graphical models, causal reasoning, Bayesian nonparametrics, and
probabilistic programming (Chomsky, 1957; Nilsson and Fikes, 1970; Pearl, 1986, 2009; Russell and
Norvig, 2009; Hjort et al., 2010; Goodman et al., 2012; Ghahramani, 2015). Entire sub-elds have
focused on explicit entity- and relation-centric learning, such as relational reinforcement learning
(D zeroski et al., 2001) and statistical relational learning (Getoor and Taskar, 2007). A key reason
why structured approaches were so vital to machine learning in previous eras was, in part, because
data and computing resources were expensive, and the improved sample complexity aorded by
structured approaches' strong inductive biases was very valuable.


data and computing resources were expensive, and the improved sample complexity aorded by
structured approaches' strong inductive biases was very valuable.
In contrast with past approaches in AI, modern deep learning methods (LeCun et al., 2015;
Schmidhuber, 2015; Goodfellow et al., 2016) often follow an \end-to-end" design philosophy which
emphasizes minimal a priori representational and computational assumptions, and seeks to avoid
explicit structure and \hand-engineering". This emphasis has t well with|and has perhaps been
armed by|the current abundance of cheap data and cheap computing resources, which make
1Whether this entails a \language of thought" (Fodor, 1975) is beyond the scope of this work.
2
trading o sample eciency for more
exible learning a rational choice. The remarkable and rapid
advances across many challenging domains, from image classication (Krizhevsky et al., 2012;
Szegedy et al., 2017), to natural language processing (Sutskever et al., 2014; Bahdanau et al., 2015),
to game play (Mnih et al., 2015; Silver et al., 2016; Morav c k et al., 2017), are a testament to this


to game play (Mnih et al., 2015; Silver et al., 2016; Morav c k et al., 2017), are a testament to this
minimalist principle. A prominent example is from language translation, where sequence-to-sequence
approaches (Sutskever et al., 2014; Bahdanau et al., 2015) have proven very eective without using
explicit parse trees or complex relationships between linguistic entities.
Despite deep learning's successes, however, important critiques (Marcus, 2001; Shalev-Shwartz
et al., 2017; Lake et al., 2017; Lake and Baroni, 2018; Marcus, 2018a,b; Pearl, 2018; Yuille and
Liu, 2018) have highlighted key challenges it faces in complex language and scene understanding,
reasoning about structured data, transferring learning beyond the training conditions, and learning
from small amounts of experience. These challenges demand combinatorial generalization, and so it
is perhaps not surprising that an approach which eschews compositionality and explicit structure
struggles to meet them.
When deep learning's connectionist (Rumelhart et al., 1987) forebears were faced with analogous
critiques from structured, symbolic positions (Fodor and Pylyshyn, 1988; Pinker and Prince, 1988),


critiques from structured, symbolic positions (Fodor and Pylyshyn, 1988; Pinker and Prince, 1988),
there was a constructive eort (Bobrow and Hinton, 1990; Marcus, 2001) to address the challenges
directly and carefully. A variety of innovative sub-symbolic approaches for representing and reasoning
about structured objects were developed in domains such as analogy-making, linguistic analysis,
symbol manipulation, and other forms of relational reasoning (Smolensky, 1990; Hinton, 1990;
Pollack, 1990; Elman, 1991; Plate, 1995; Eliasmith, 2013), as well as more integrative theories for
how the mind works (Marcus, 2001). Such work also helped cultivate more recent deep learning
advances which use distributed, vector representations to capture rich semantic content in text
(Mikolov et al., 2013; Pennington et al., 2014), graphs (Narayanan et al., 2016, 2017), algebraic and
logical expressions (Allamanis et al., 2017; Evans et al., 2018), and programs (Devlin et al., 2017;
Chen et al., 2018b).
We suggest that a key path forward for modern AI is to commit to combinatorial generalization
as a top priority, and we advocate for integrative approaches to realize this goal. Just as biology does


as a top priority, and we advocate for integrative approaches to realize this goal. Just as biology does
not choose between nature versus nurture|it uses nature and nurture jointly , to build wholes which
are greater than the sums of their parts|we, too, reject the notion that structure and
exibility are
somehow at odds or incompatible, and embrace both with the aim of reaping their complementary
strengths. In the spirit of numerous recent examples of principled hybrids of structure-based methods
and deep learning (e.g., Reed and De Freitas, 2016; Garnelo et al., 2016; Ritchie et al., 2016; Wu
et al., 2017; Denil et al., 2017; Hudson and Manning, 2018), we see great promise in synthesizing
new techniques by drawing on the full AI toolkit and marrying the best approaches from today
with those which were essential during times when data and computation were at a premium.
Recently, a class of models has arisen at the intersection of deep learning and structured
approaches, which focuses on approaches for reasoning about explicitly structured data, in particular
graphs (e.g. Scarselli et al., 2009b; Bronstein et al., 2017; Gilmer et al., 2017; Wang et al., 2018c; Li
et al., 2018; Kipf et al., 2018; Gulcehre et al., 2018). What these approaches all have in common


et al., 2018; Kipf et al., 2018; Gulcehre et al., 2018). What these approaches all have in common
is a capacity for performing computation over discrete entities and the relations between them.
What sets them apart from classical approaches is how the representations and structure of the
entities and relations|and the corresponding computations|can be learned, relieving the burden
of needing to specify them in advance. Crucially, these methods carry strong relational inductive
biases , in the form of specic architectural assumptions, which guide these approaches towards
learning about entities and relations (Mitchell, 1980), which we, joining many others (Spelke et al.,
1992; Spelke and Kinzler, 2007; Marcus, 2001; Tenenbaum et al., 2011; Lake et al., 2017; Lake and
Baroni, 2018; Marcus, 2018b), suggest are an essential ingredient for human-like intelligence.
3
Box 1: Relational reasoning
We dene structure as the product of composing a set of known building blocks. \Structured
representations" capture this composition (i.e., the arrangement of the elements) and \structured
computations" operate over the elements and their composition as a whole. Relational reasoning,
then, involves manipulating structured representations of entities and relations , using rules


computations" operate over the elements and their composition as a whole. Relational reasoning,
then, involves manipulating structured representations of entities and relations , using rules
for how they can be composed. We use these terms to capture notions from cognitive science,
theoretical computer science, and AI, as follows:
Anentity is an element with attributes, such as a physical object with a size and mass.
Arelation is a property between entities. Relations between two objects might include
same size as ,heavier than , and distance from . Relations can have attributes as
well. The relation more than Xtimes heavier than takes an attribute, X, which
determines the relative weight threshold for the relation to be true vs.false . Relations
can also be sensitive to the global context. For a stone and a feather, the relation falls
with greater acceleration than depends on whether the context is in air vs.in a
vacuum . Here we focus on pairwise relations between entities.
Aruleis a function (like a non-binary logical predicate) that maps entities and relations
to other entities and relations, such as a scale comparison like is entity X large? and
is entity X heavier than entity Y? . Here we consider rules which take one or two
arguments (unary and binary), and return a unary property value.


is entity X heavier than entity Y? . Here we consider rules which take one or two
arguments (unary and binary), and return a unary property value.
As an illustrative example of relational reasoning in machine learning, graphical models (Pearl,
1988; Koller and Friedman, 2009) can represent complex joint distributions by making explicit
random conditional independences among random variables. Such models have been very
successful because they capture the sparse structure which underlies many real-world generative
processes and because they support ecient algorithms for learning and reasoning. For example,
hidden Markov models constrain latent states to be conditionally independent of others given
the state at the previous time step, and observations to be conditionally independent given the
latent state at the current time step, which are well-matched to the relational structure of many
real-world causal processes. Explicitly expressing the sparse dependencies among variables
provides for various ecient inference and reasoning algorithms, such as message-passing, which
apply a common information propagation procedure across localities within a graphical model,
resulting in a composable, and partially parallelizable, reasoning procedure which can be applied
to graphical models of dierent sizes and shape.
In the remainder of the paper, we examine various deep learning methods through the lens of
their relational inductive biases, showing that existing methods often carry relational assumptions


In the remainder of the paper, we examine various deep learning methods through the lens of
their relational inductive biases, showing that existing methods often carry relational assumptions
which are not always explicit or immediately evident. We then present a general framework for
entity- and relation-based reasoning|which we term graph networks |for unifying and extending
existing methods which operate on graphs, and describe key design principles for building powerful
architectures using graph networks as building blocks. We have also released an open-source library
for building graph networks, which can be found here: github.com/deepmind/graph nets .
2 Relational inductive biases
Many approaches in machine learning and AI which have a capacity for relational reasoning
4
Box 2: Inductive biases
Learning is the process of apprehending useful knowledge by observing and interacting with the
world. It involves searching a space of solutions for one expected to provide a better explanation
of the data or to achieve higher rewards. But in many cases, there are multiple solutions which
are equally good (Goodman, 1955). An inductive bias allows a learning algorithm to prioritize
one solution (or interpretation) over another, independent of the observed data (Mitchell,
1980). In a Bayesian model, inductive biases are typically expressed through the choice and
parameterization of the prior distribution (Griths et al., 2010). In other contexts, an inductive


parameterization of the prior distribution (Griths et al., 2010). In other contexts, an inductive
bias might be a regularization term (McClelland, 1994) added to avoid overtting, or it might
be encoded in the architecture of the algorithm itself. Inductive biases often trade
exibility
for improved sample complexity and can be understood in terms of the bias-variance tradeo
(Geman et al., 1992). Ideally, inductive biases both improve the search for solutions without
substantially diminishing performance, as well as help nd solutions which generalize in a
desirable way; however, mismatched inductive biases can also lead to suboptimal performance
by introducing constraints that are too strong.
Inductive biases can express assumptions about either the data-generating process or the space
of solutions. For example, when tting a 1D function to data, linear least squares follows
the constraint that the approximating function be a linear model, and approximation errors
should be minimal under a quadratic penalty. This re
ects an assumption that the datagenerating process can be explained simply, as a line process corrupted by additive Gaussian
noise. Similarly, L2 regularization prioritizes solutions whose parameters have small values,
and can induce unique solutions and global structure to otherwise ill-posed problems. This can


noise. Similarly, L2 regularization prioritizes solutions whose parameters have small values,
and can induce unique solutions and global structure to otherwise ill-posed problems. This can
be interpreted as an assumption about the learning process: that searching for good solutions
is easier when there is less ambiguity among solutions. Note, these assumptions need not be
explicit|they re
ect interpretations of how a model or algorithm interfaces with the world.
(Box 1) use a relational inductive bias . While not a precise, formal denition, we use this term to
refer generally to inductive biases (Box 2) which impose constraints on relationships and interactions
among entities in a learning process.
Creative new machine learning architectures have rapidly proliferated in recent years, with
(perhaps not surprisingly given the thesis of this paper) practitioners often following a design pattern
of composing elementary building blocks to form more complex, deep2computational hierarchies and
graphs3. Building blocks such as \fully connected" layers are stacked into \multilayer perceptrons"
(MLPs), \convolutional layers" are stacked into \convolutional neural networks" (CNNs), and a
standard recipe for an image processing network is, generally, some variety of CNN composed with
a MLP. This composition of layers provides a particular type of relational inductive bias|that


standard recipe for an image processing network is, generally, some variety of CNN composed with
a MLP. This composition of layers provides a particular type of relational inductive bias|that
of hierarchical processing|in which computations are performed in stages, typically resulting in
increasingly long range interactions among information in the input signal. As we explore below, the
building blocks themselves also carry various relational inductive biases (Table 1). Though beyond
the scope of this paper, various non-relational inductive biases are used in deep learning as well: for
example, activation non-linearities, weight decay, dropout (Srivastava et al., 2014), batch and layer
normalization (Ioe and Szegedy, 2015; Ba et al., 2016), data augmentation, training curricula, and
optimization algorithms all impose constraints on the trajectory and outcome of learning.
2This pattern of composition in depth is ubiquitous in deep learning, and is where the \deep" comes from.
3Recent methods (Liu et al., 2018) even automate architecture construction via learned graph editing procedures.
5
Component Entities Relations Rel. inductive bias Invariance
Fully connected Units All-to-all Weak Convolutional Grid elements Local Locality Spatial translation
Recurrent Timesteps Sequential Sequentiality Time translation
Graph network Nodes Edges Arbitrary Node, edge permutations


Recurrent Timesteps Sequential Sequentiality Time translation
Graph network Nodes Edges Arbitrary Node, edge permutations
Table 1: Various relational inductive biases in standard deep learning components. See also Section 2.
To explore the relational inductive biases expressed within various deep learning methods, we
must identify several key ingredients, analogous to those in Box 1: what are the entities , what are
therelations , and what are the rules for composing entities and relations, and computing their
implications? In deep learning, the entities and relations are typically expressed as distributed
representations, and the rules as neural network function approximators; however, the precise forms
of the entities, relations, and rules vary between architectures. To understand these dierences
between architectures, we can further ask how each supports relational reasoning by probing:
The arguments to the rule functions (e.g., which entities and relations are provided as input).
How the rule function is reused , orshared , across the computational graph (e.g., across dierent
entities and relations, across dierent time or processing steps, etc.).
How the architecture denes interactions versus isolation among representations (e.g., by
applying rules to draw conclusions about related entities, versus processing them separately).
2.1 Relational inductive biases in standard deep learning building blocks
2.1.1 Fully connected layers


applying rules to draw conclusions about related entities, versus processing them separately).
2.1 Relational inductive biases in standard deep learning building blocks
2.1.1 Fully connected layers
Perhaps the most common building block is a fully connected layer (Rosenblatt, 1961). Typically
implemented as a non-linear vector-valued function of vector inputs, each element, or \unit", of
the output vector is the dot product between a weight vector, followed by an added bias term, and
nally a non-linearity such as a rectied linear unit (ReLU). As such, the entities are the units in
the network, the relations are all-to-all (all units in layer iare connected to all units in layer j),
and the rules are specied by the weights and biases. The argument to the rule is the full input
signal, there is no reuse, and there is no isolation of information (Figure 1a). The implicit relational
inductive bias in a fully connected layer is thus very weak: all input units can interact to determine
any output unit's value, independently across outputs (Table 1).
2.1.2 Convolutional layers
Another common building block is a convolutional layer (Fukushima, 1980; LeCun et al., 1989). It is


2.1.2 Convolutional layers
Another common building block is a convolutional layer (Fukushima, 1980; LeCun et al., 1989). It is
implemented by convolving an input vector or tensor with a kernel of the same rank, adding a bias
term, and applying a point-wise non-linearity. The entities here are still individual units (or grid
elements, e.g. pixels), but the relations are sparser. The dierences between a fully connected layer
and a convolutional layer impose some important relational inductive biases: locality and translation
invariance (Figure 1b). Locality re
ects that the arguments to the relational rule are those entities in
close proximity with one another in the input signal's coordinate space, isolated from distal entities.
Translation invariance re
ects reuse of the same rule across localities in the input. These biases
are very eective for processing natural image data because there is high covariance within local
6
(a) Fully connected
Sharing in space (b) Convolutional
Sharing in time (c) Recurrent
Figure 1: Reuse and sharing in common deep learning building blocks. (a) Fully connected layer,
in which all weights are independent, and there is no sharing. (b) Convolutional layer, in which


in which all weights are independent, and there is no sharing. (b) Convolutional layer, in which
a local kernel function is reused multiple times across the input. Shared weights are indicated by
arrows with the same color. (c) Recurrent layer, in which the same function is reused across dierent
processing steps.
neighborhoods, which diminishes with distance, and because the statistics are mostly stationary
across an image (Table 1).
2.1.3 Recurrent layers
A third common building block is a recurrent layer (Elman, 1990), which is implemented over a
sequence of steps. Here, we can view the inputs and hidden states at each processing step as the
entities, and the Markov dependence of one step's hidden state on the previous hidden state and
the current input, as the relations. The rule for combining the entities takes a step's inputs and
hidden state as arguments to update the hidden state. The rule is reused over each step (Figure 1c),
which re
ects the relational inductive bias of temporal invariance (similar to a CNN's translational
invariance in space). For example, the outcome of some physical sequence of events should not
depend on the time of day. RNNs also carry a bias for locality in the sequence via their Markovian
structure (Table 1).


depend on the time of day. RNNs also carry a bias for locality in the sequence via their Markovian
structure (Table 1).
2.2 Computations over sets and graphs
While the standard deep learning toolkit contains methods with various forms of relational inductive
biases, there is no \default" deep learning component which operates on arbitrary relational structure.
We need models with explicit representations of entities and relations, and learning algorithms which
nd rules for computing their interactions, as well as ways of grounding them in data. Importantly,
entities in the world (such as objects and agents) do not have a natural order; rather, orderings
can be dened by the properties of their relations. For example, the relations between the sizes of
a set of objects can potentially be used to order them, as can their masses, ages, toxicities, and
prices. Invariance to ordering|except in the face of relations|is a property that should ideally be
re
ected by a deep learning component for relational reasoning.
Sets are a natural representation for systems which are described by entities whose order is
undened or irrelevant; in particular, their relational inductive bias does not come from the presence
of something, but rather from the absence . For illustration, consider the task of predicting the center
7
H H
O
The brown
dog jumped. The


of something, but rather from the absence . For illustration, consider the task of predicting the center
7
H H
O
The brown
dog jumped. The
brown dog jumped (a) (b)
(c) (d)
(e) (f)
Molecule Mass-Spring System
n-body System Rigid Body System
Sentence and Parse Tree Image and Fully-Connected Scene Graph Figure 2: Dierent graph representations. (a) A molecule, in which each atom is represented as a
node and edges correspond to bonds (e.g. Duvenaud et al., 2015). (b) A mass-spring system, in
which the rope is dened by a sequence of masses which are represented as nodes in the graph (e.g.
Battaglia et al., 2016; Chang et al., 2017). (c) A n-body system, in which the bodies are nodes and
the underlying graph is fully connected (e.g. Battaglia et al., 2016; Chang et al., 2017). (d) A rigid
body system, in which the balls and walls are nodes, and the underlying graph denes interactions
between the balls and between the balls and the walls (e.g. Battaglia et al., 2016; Chang et al., 2017).


between the balls and between the balls and the walls (e.g. Battaglia et al., 2016; Chang et al., 2017).
(e) A sentence, in which the words correspond to leaves in a tree, and the other nodes and edges
could be provided by a parser (e.g. Socher et al., 2013). Alternately, a fully connected graph could
be used (e.g. Vaswani et al., 2017). (f) An image, which can be decomposed into image patches
corresponding to nodes in a fully connected graph (e.g. Santoro et al., 2017; Wang et al., 2018c).
of mass of a solar system comprised of nplanets, whose attributes (e.g., mass, position, velocity,
etc.) are denoted by fx1;x2;:::;xng. For such a computation, the order in which we consider the
planets does not matter because the state can be described solely in terms of aggregated, averaged
quantities. However, if we were to use a MLP for this task, having learned the prediction for a
particular input ( x1;x2;:::;xn) would not necessarily transfer to making a prediction for the same


particular input ( x1;x2;:::;xn) would not necessarily transfer to making a prediction for the same
inputs under a dierent ordering ( xn;x1;:::;x2). Since there are n! such possible permutations, in
the worst case, the MLP could consider each ordering as fundamentally dierent, and thus require
an exponential number of input/output training examples to learn an approximating function.
A natural way to handle such combinatorial explosion is to only allow the prediction to depend
on symmetric functions of the inputs' attributes. This might mean computing shared per-object
8
featuresff(x1);:::;f (xn)gwhich are then aggregated in a symmetric way (for example, by taking
their mean). Such an approach is the essence of the Deep Sets and related models (Zaheer et al.,
2017; Edwards and Storkey, 2016; Pevn y and Somol, 2017), which we explore further in Section 4.2.3.
Of course, permutation invariance is not the only important form of underlying structure in
many problems. For example, each object in a set may be aected by pairwise interactions with
the other objects in the set (Hartford et al., 2018). In our planets scenario, consider now the


the other objects in the set (Hartford et al., 2018). In our planets scenario, consider now the
task of predicting each individual planet's position after a time interval,  t. In this case, using
aggregated, averaged information is not enough because the movement of each planet depends on
the forces the other planets are exerting on it. Instead, we could compute the state of each object
asx0
i=f(xi;P
jg(xi;xj)), wheregcould compute the force induced by the j-th planet on the
i-th planet, and fcould compute the future state of the i-th planet which results from the forces
and dynamics. The fact that we use the same geverywhere is again a consequence of the global
permutation invariance of the system; however, it also supports a dierent relational structure
becausegnow takes two arguments rather than one.4
The above solar system examples illustrate two relational structures: one in which there are
no relations, and one which consists of all pairwise relations. Many real-world systems (such as
in Figure 2) have a relational structure somewhere in between these two extremes, however, with
some pairs of entities possessing a relation and others lacking one. In our solar system example, if
the system instead consists of the planets and their moons, one may be tempted to approximate


some pairs of entities possessing a relation and others lacking one. In our solar system example, if
the system instead consists of the planets and their moons, one may be tempted to approximate
it by neglecting the interactions between moons of dierent planets. In practice, this means
computing interactions only between some pairs of objects, i.e. x0
i=f(xi;P
j2(i)g(xi;xj)), where
(i)f1;:::;ngis a neighborhood around node i. This corresponds to a graph, in that the i-th
object only interacts with a subset of the other objects, described by its neighborhood. Note, the
updated states still do not depend in the order in which we describe the neighborhood.5
Graphs, generally, are a representation which supports arbitrary (pairwise) relational structure, and computations over graphs aord a strong relational inductive bias beyond that which
convolutional and recurrent layers can provide.
3 Graph networks
Neural networks that operate on graphs, and structure their computations accordingly, have been
developed and explored extensively for more than a decade under the umbrella of \graph neural
networks" (Gori et al., 2005; Scarselli et al., 2005, 2009a; Li et al., 2016), but have grown rapidly


networks" (Gori et al., 2005; Scarselli et al., 2005, 2009a; Li et al., 2016), but have grown rapidly
in scope and popularity in recent years. We survey the literature on these methods in the next
sub-section (3.1). Then in the remaining sub-sections, we present our graph networks framework,
which generalizes and extends several lines of work in this area.
3.1 Background
Models in the graph neural network family (Gori et al., 2005; Scarselli et al., 2005, 2009a; Li et al.,
2016) have been explored in a diverse range of problem domains, across supervised, semi-supervised,
unsupervised, and reinforcement learning settings. They have been eective at tasks thought to
have rich relational structure, such as visual scene understanding tasks (Raposo et al., 2017; Santoro
4We could extend this same analysis to increasingly entangled structures that depend on relations among triplets
(i.e.,g(xi;xj;xk)), quartets, and so on. We note that if we restrict these functions to only operate on subsets of xi
which are spatially close, then we end back up with something resembling CNNs. In the most entangled sense, where


which are spatially close, then we end back up with something resembling CNNs. In the most entangled sense, where
there is a single relation function g(x1; : : : ;xn), we end back up with a construction similar to a fully connected layer.
5The invariance which this model enforces is the invariance under isomorphism of the graph.
9
et al., 2017) and few-shot learning (Garcia and Bruna, 2018). They have also been used to learn
the dynamics of physical systems (Battaglia et al., 2016; Chang et al., 2017; Watters et al., 2017;
van Steenkiste et al., 2018; Sanchez-Gonzalez et al., 2018) and multi-agent systems (Sukhbaatar
et al., 2016; Hoshen, 2017; Kipf et al., 2018), to reason about knowledge graphs (Bordes et al., 2013;
O~ noro-Rubio et al., 2017; Hamaguchi et al., 2017), to predict the chemical properties of molecules
(Duvenaud et al., 2015; Gilmer et al., 2017), to predict trac on roads (Li et al., 2017; Cui et al.,
2018), to classify and segment images and videos (Wang et al., 2018c; Hu et al., 2017) and 3D


2018), to classify and segment images and videos (Wang et al., 2018c; Hu et al., 2017) and 3D
meshes and point clouds (Wang et al., 2018d), to classify regions in images (Chen et al., 2018a), to
perform semi-supervised text classication (Kipf and Welling, 2017), and in machine translation
(Vaswani et al., 2017; Shaw et al., 2018; Gulcehre et al., 2018). They have been used within both
model-free (Wang et al., 2018b) and model-based (Hamrick et al., 2017; Pascanu et al., 2017;
Sanchez-Gonzalez et al., 2018) continuous control, for model-free reinforcement learning (Hamrick
et al., 2018; Zambaldi et al., 2018), and for more classical approaches to planning (Toyer et al.,
2017).
Many traditional computer science problems, which involve reasoning about discrete entities and
structure, have also been explored with graph neural networks, such as combinatorial optimization
(Bello et al., 2016; Nowak et al., 2017; Dai et al., 2017), boolean satisability (Selsam et al., 2018),
program representation and verication (Allamanis et al., 2018; Li et al., 2016), modeling cellular


program representation and verication (Allamanis et al., 2018; Li et al., 2016), modeling cellular
automata and Turing machines (Johnson, 2017), and performing inference in graphical models
(Yoon et al., 2018). Recent work has also focused on building generative models of graphs (Li et al.,
2018; De Cao and Kipf, 2018; You et al., 2018; Bojchevski et al., 2018), and unsupervised learning of
graph embeddings (Perozzi et al., 2014; Tang et al., 2015; Grover and Leskovec, 2016; Garc a-Dur an
and Niepert, 2017).
The works cited above are by no means an exhaustive list, but provide a representative crosssection of the breadth of domains for which graph neural networks have proven useful. We point
interested readers to a number of existing reviews which examine the body of work on graph neural
networks in more depth. In particular, Scarselli et al. (2009a) provides an authoritative overview of
early graph neural network approaches. Bronstein et al. (2017) provides an excellent survey of deep
learning on non-Euclidean data, and explores graph neural nets, graph convolution networks, and
related spectral approaches. Recently, Gilmer et al. (2017) introduced the message-passing neural


related spectral approaches. Recently, Gilmer et al. (2017) introduced the message-passing neural
network (MPNN), which unied various graph neural network and graph convolutional network
approaches (Monti et al., 2017; Bruna et al., 2014; Hena et al., 2015; Deerrard et al., 2016;
Niepert et al., 2016; Kipf and Welling, 2017; Bronstein et al., 2017) by analogy to message-passing
in graphical models. In a similar vein, Wang et al. (2018c) introduced the non-local neural network
(NLNN), which unied various \self-attention"-style methods (Vaswani et al., 2017; Hoshen, 2017;
Veli ckovi c et al., 2018) by analogy to methods from computer vision and graphical models for
capturing long range dependencies in signals.
3.2 Graph network (GN) block
We now present our graph networks (GN) framework, which denes a class of functions for
relational reasoning over graph-structured representations. Our GN framework generalizes and
extends various graph neural network, MPNN, and NLNN approaches (Scarselli et al., 2009a; Gilmer
et al., 2017; Wang et al., 2018c), and supports constructing complex architectures from simple


et al., 2017; Wang et al., 2018c), and supports constructing complex architectures from simple
building blocks. Note, we avoided using the term \neural" in the \graph network" label to re
ect
that they can be implemented with functions other than neural networks, though here our focus is
on neural network implementations.
The main unit of computation in the GN framework is the GN block , a \graph-to-graph" module
10
Box 3: Our denition of \graph"
Attributesviek


Here we use \graph" to mean a directed, attributed multi-graph with a global attribute. In our
terminology, a node is denoted as vi, an edge as ek, and the global attributes as u. We also use
skandrkto indicate the indices of the sender and receiver nodes (see below), respectively, for
edgek. To be more precise, we dene these terms as:
Directed : one-way edges, from a \sender" node to a \receiver" node.
Attribute : properties that can be encoded as a vector, set, or even another graph.
Attributed : edges and vertices have attributes associated with them.
Global attribute : a graph-level attribute.
Multi-graph : there can be more than one edge between vertices, including self-edges.
Figure 2 shows a variety of dierent types of graphs corresponding to real data that we may be
interested in modeling, including physical systems, molecules, images, and text.
which takes a graph as input, performs computations over the structure, and returns a graph as
output. As described in Box 3, entities are represented by the graph's nodes , relations by the edges ,
and system-level properties by global attributes. The GN framework's block organization emphasizes
customizability and synthesizing new architectures which express desired relational inductive biases.


and system-level properties by global attributes. The GN framework's block organization emphasizes
customizability and synthesizing new architectures which express desired relational inductive biases.
The key design principles are: Flexible representations (see Section 4.1); Congurable within-block
structure (see Section 4.2); and Composable multi-block architectures (see Section 4.3).
We introduce a motivating example to help make the GN formalism more concrete. Consider
predicting the movements a set of rubber balls in an arbitrary gravitational eld, which, instead of
bouncing against one another, each have one or more springs which connect them to some (or all) of
the others. We will refer to this running example throughout the denitions below, to motivate the
graph representation and the computations operating over it. Figure 2 depicts some other common
scenarios that can be represented by graphs and reasoned over using graph networks.
3.2.1 Denition of \graph"
Within our GN framework, a graph is dened as a 3-tuple G=(u;V;E )(see Box 3 for details of
graph representations). The uis a global attribute; for example, umight represent the gravitational


graph representations). The uis a global attribute; for example, umight represent the gravitational
eld. TheV=fvigi=1:Nvis the set of nodes (of cardinality Nv), where each viis a node's attribute.
For example, Vmight represent each ball, with attributes for position, velocity, and mass. The
E=f(ek;rk;sk)gk=1:Neis the set of edges (of cardinality Ne), where each ekis the edge's attribute,
rkis the index of the receiver node, and skis the index of the sender node. For example, Emight
represent the presence of springs between dierent balls, and their corresponding spring constants.
11
Algorithm 1 Steps of computation in a full GN block.
function GraphNetwork (E,V,u)
fork2f1:::Negdo
e0
k e(ek;vrk;vsk;u) .1. Compute updated edge attributes
end for
fori2f1:::Nngdo
letE0
i=f(e0
k;rk;sk)grk=i;k=1:Ne
 e0
i e!v(E0
i) .2. Aggregate edge attributes per node
v0


 e0
i e!v(E0
i) .2. Aggregate edge attributes per node
v0
i v( e0
i;vi;u) .3. Compute updated node attributes
end for
letV0=fv0gi=1:Nv
letE0=f(e0
k;rk;sk)gk=1:Ne
 e0 e!u(E0) .4. Aggregate edge attributes globally
 v0 v!u(V0) .5. Aggregate node attributes globally
u0 u( e0; v0;u) .6. Compute updated global attribute
return (E0;V0;u0)
end function
3.2.2 Internal structure of a GN block
A GN block contains three \update" functions, , and three \aggregation" functions, ,
e0
k=e(ek;vrk;vsk;u)
v0
i=v
 e0
i;vi;u
u0=u
 e0; v0;u e0
i=e!v
E0
i
 e0=e!u
E0


i=e!v
E0
i
 e0=e!u
E0
 v0=v!u
V0(1)
whereE0
i=f(e0
k;rk;sk)grk=i;k=1:Ne,V0=fv0
igi=1:Nv, andE0=S
iE0
i=f(e0
k;rk;sk)gk=1:Ne.
Theeis mapped across all edges to compute per-edge updates, the vis mapped across all
nodes to compute per-node updates, and the uis applied once as the global update. The
functions each take a set as input, and reduce it to a single element which represents the aggregated
information. Crucially, the functions must be invariant to permutations of their inputs, and should
take variable numbers of arguments (e.g., elementwise summation, mean, maximum, etc.).
3.2.3 Computational steps within a GN block
When a graph, G, is provided as input to a GN block, the computations proceed from the edge, to
the node, to the global level. Figure 3 shows a depiction of which graph elements are involved in


the node, to the global level. Figure 3 shows a depiction of which graph elements are involved in
each of these computations, and Figure 4a shows a full GN block, with its update and aggregation
functions. Algorithm 1 shows the following steps of computation:
1.eis applied per edge, with arguments ( ek;vrk;vsk;u), and returns e0
k. In our springs example,
this might correspond to the forces or potential energies between two connected balls. The
set of resulting per-edge outputs for each node, i, is,E0
i=f(e0
k;rk;sk)grk=i;k=1:Ne. And
E0=S
iE0
i=f(e0
k;rk;sk)gk=1:Neis the set of all per-edge outputs.
2.e!vis applied to E0
i, and aggregates the edge updates for edges that project to vertex i, into
 e0
i, which will be used in the next step's node update. In our running example, this might
correspond to summing all the forces or potential energies acting on the ithball.
12
vi


Figure 3: Updates in a GN block. Blue indicates the element that is being updated, and black
indicates other elements which are involved in the update (note that the pre-update value of the
blue element is also used in the update). See Equation 1 for details on the notation.
3.vis applied to each node i, to compute an updated node attribute, v0
i. In our running
example,vmay compute something analogous to the updated position, velocity, and kinetic
energy of each ball. The set of resulting per-node outputs is, V0=fv0
igi=1:Nv.
4.e!uis applied to E0, and aggregates all edge updates, into  e0, which will then be used in the
next step's global update. In our running example, e!umay compute the summed forces
(which should be zero, in this case, due to Newton's third law) and the springs' potential
energies.
5.v!uis applied to V0, and aggregates all node updates, into  v0, which will then be used in
the next step's global update. In our running example, v!umight compute the total kinetic
energy of the system.


the next step's global update. In our running example, v!umight compute the total kinetic
energy of the system.
6.uis applied once per graph, and computes an update for the global attribute, u0. In our
running example, umight compute something analogous to the net forces and total energy
of the physical system.
Note, though we assume this sequence of steps here, the order is not strictly enforced: it is possible
to reverse the update functions to proceed from global, to per-node, to per-edge updates, for example.
Kearnes et al. (2016) computes edge updates from nodes in a similar manner.
3.2.4 Relational inductive biases in graph networks
Our GN framework imposes several strong relational inductive biases when used as components in
a learning process. First, graphs can express arbitrary relationships among entities, which means
the GN's input determines how representations interact and are isolated, rather than those choices
being determined by the xed architecture. For example, the assumption that two entities have a
relationship|and thus should interact|is expressed by an edge between the entities' corresponding
nodes. Similarly, the absence of an edge expresses the assumption that the nodes have no relationship
and should not in
uence each other directly.


nodes. Similarly, the absence of an edge expresses the assumption that the nodes have no relationship
and should not in
uence each other directly.
Second, graphs represent entities and their relations as sets, which are invariant to permutations.
This means GNs are invariant to the order of these elements6, which is often desirable. For example,
the objects in a scene do not have a natural ordering (see Sec. 2.2).
Third, a GN's per-edge and per-node functions are reused across all edges and nodes, respectively.
This means GNs automatically support a form of combinatorial generalization (see Section 5.1):
because graphs are composed of edges, nodes, and global features, a single GN can operate on
graphs of dierent sizes (numbers of edges and nodes) and shapes (edge connectivity).
6Note, an ordering can be imposed by encoding the indices in the node or edge attributes, or via the edges
themselves (e.g. by encoding a chain or partial ordering).
13
4 Design principles for graph network architectures
The GN framework can be used to implement a wide variety of architectures, in accordance with
the design principles listed above in Section 3.2, which also correspond to the sub-sections (4.1,


the design principles listed above in Section 3.2, which also correspond to the sub-sections (4.1,
4.2, and 4.3) below. In general, the framework is agnostic to specic attribute representations and
functional forms. Here, however, we focus mainly on deep learning architectures, which allow GNs
to act as learnable graph-to-graph function approximators.
4.1 Flexible representations
Graph networks support highly
exible graph representations in two ways: rst, in terms of the
representation of the attributes; and second, in terms of the structure of the graph itself.
4.1.1 Attributes
The global, node, and edge attributes of a GN block can use arbitrary representational formats. In
deep learning implementations, real-valued vectors and tensors are most common. However, other
data structures such as sequences, sets, or even graphs could also be used.
The requirements of the problem will often determine what representations should be used for
the attributes. For example, when the input data is an image, the attributes might be represented
as tensors of image patches; however, when the input data is a text document, the attributes might
be sequences of words corresponding to sentences.
For each GN block within a broader architecture, the edge and node outputs typically correspond