Fig. 2. A functional overview of the model. A global convolutional model extracts
features and splits into two heads trained to predict, respectively activity classes and
object classes. The latter are predicted by pooling over object instance masks, which
are predicted by an additional convolutional model. The object instances are passed
through a visual reasoning module.
is a CNN-based architecture employing convolutional layers, including spatiotemporal convolutions, able to extract global motion features. However, it is not
able to extract information from an object level perspective. We leverage the
object head to perform reasoning on the relationships between predicted object
instances.
Our main contribution is a new structured module called Object Relation
Network (ORN), which is able to perform spatio-temporal reasoning between
detected object instances in the video. ORN is able to reason by modeling how
objects move, appear and disappear and how they interact between two frames.
In this section, we will rst describe our main contribution, the ORN network.
We then provide details about object instance features, about the activity head,
and nally about the nal recognition task. In what follows, lowercase letters
denote 1D vectors while uppercase letters are used for 2D and 3D matrices or
higher order tensors. We assume that the input of our system is a video of T

Object Level Visual Reasoning in Videos


denote 1D vectors while uppercase letters are used for 2D and 3D matrices or
higher order tensors. We assume that the input of our system is a video of T
frames denoted by X1:T= (Xt)T
t=1where Xtis the RGB image at timestep t.
The goal is to learn a mapping from X1:Tto activity classes y.
3.1 Object Relation Network
ORN (Object Relation Network) is a module for reasoning between semantic
objects through space and time. It captures object moves, arrivals and interactions in an ecient manner. We suppose that for each frame t, we have a set
of objects kwith associated features ok
t. Objects and features are detected and
computed by the object head described in Section 3.2.
Reasoning about activities in videos is inherently temporal, as activities follow the arrow of time [28], i.e. the causality of the time dimension imposes that
past actions have consequences in the future but notvice-versa. We handle this
by sampling: running a process over time t, and for each instant t, sampling a
Object Level Visual Reasoning in Videos 7
second frame t0with t0<t. Our network reasons on objects which interact between pairs of frames and their corresponding sets of objects Ot0=
ok
t0	K0

Object Level Visual Reasoning in Videos

Summary:
Human activity recognition is typically addressed by detecting key concepts
like global and local motion, features related to object classes present in the
scene, as well as features related to the global context. The next open
challenges in activity recognition require a level of understanding that pushes
beyond this and call for models with capabilities for fine distinction and
detailed comprehension of interactions between actors and objects in a scene.
We propose a model capable of learning to reason about semantically meaningful
spatiotemporal interactions in videos. The key to our approach is a choice of
performing this reasoning at the object level through the integration of state
of the art object detection networks. This allows the model to learn detailed
spatial interactions that exist at a semantic, object-interaction relevant
level. We evaluate our method on three standard datasets (Twenty-BN
Something-Something, VLOG and EPIC Kitchens) and achieve state of the art
results on all of them. Finally, we show visualizations of the interactions
learned by the model, which illustrate object classes and their interactions
corresponding to different activity classes.

Fabien Baradel
Natalia Neverova
Christian Wolf
Julien Mille
Greg Mori

Accepted at ECCV 2018 - long version (16 pages + ref)


second frame t0with t0<t. Our network reasons on objects which interact between pairs of frames and their corresponding sets of objects Ot0=
ok
t0	K0
k=1
andOt=
ok
t	K
k=1. The goal is to learn a general function dened on the set of
all input objects from the combined set of both frames:
gt=g(o1
t0; : : : ;oK0
t0;o1
t; : : : ;oK
t): (1)
The objects in this set are unordered, aside for the frame they belong to.
This task is related to a problem raised in the PointNet algorithm [29] discussed in Section 2. PointNet approximates a general function gover an item
setS=fx1; x2; : : : ; xNgas a symmetric function g0on transformed elements of
the set
g(x1; x2; : : : ; xN)g0(h(x1); h(x2); : : : ; h (xN)): (2)
In [29], g0is a max pooling operation. The authors show that it allows universal

Object Level Visual Reasoning in Videos


In [29], g0is a max pooling operation. The authors show that it allows universal
approximation of continuous sets of functions given that the hidden representation (the output of the mapping h()) is of suciently high dimension.
We argue that the approximation in (2) can be extended as follows:
g(x1; x2; : : : ; xN)f X
c2Chc([i2cxi)!
(3)
whereCis the set of cliques of a graph dened over the item set S,[is the
concatenation operator and we chose the sum operator as symmetric function.
The input dimension of the non-linearity hc() depends on the size of clique c
but maps to a xed output dimension H. In the case where Cis composed of
unary cliques only, form (3) decomposes like (2) with the exception of a dierent
symmetry operator (sum instead of max pooling). Choosing dierent graphical
structures through Swill lead to dierent terms in the summation and allows
modeling dierent types of interactions between items in the item set.
Note that the interactions between items in (3) are not exclusively modeled

Object Level Visual Reasoning in Videos


modeling dierent types of interactions between items in the item set.
Note that the interactions between items in (3) are not exclusively modeled
through hc(). Indeed, it is interesting to note, that the graphical decomposition provided byCleads to interactions which are dierent from the interactions
the same decomposition would provide when used in a probabilistic graphical
model, like for instance a Markov Random Field (MRF). In particular, a decomposition into unary terms only, as given in equation (2), does notlead to
independence between items, whereas an MRF with unary terms only is equivalent to a distribution over independent random variables. This is a consequence
of the global mapping f(), which is dened on the sum over all direct interactions. Higher-order interactions between several items not directly modeled
through a non-linearity hc() can eventually be learned by the model through
the joint output space of all hc(), provided that the dimensionality Hof this
space is high enough to incorporate all interactions. However, whereas the mappinghc() provides a direct model of interactions between pairs of items, learning
interactions between two items ( j; k), which are not directly captured through a
8 Baradel et al.
visual reasoningmodule

Object Level Visual Reasoning in Videos


Object Level Visual Reasoning in Videos


Fig. 3. ORN in the object head operating on detected instances of objects.
clique cand its corresponding hc(), requires learning a corresponding subspace
in the common output space spanned by all hc().
This leads to the question of how to dene the trade-o between the complexity of the decomposition Cand the output dimensionality Hof the mapping
hc(), both of which will determine the complexity of the modeled interactions.
Increasing the size of cliques in Cwill increase the input dimension (and therefore the capacity) of the mapping hc() as well as the computational complexity
of the sum operation.
Inspired by relational networks [32], we chose to directly model inter-frame
interactions between pairs of objects ( j; k) and leave modeling of higher-order
interactions to the output space of the mappings hand the global mapping f:
gt=X
j;kh(oj
t0;ok
t) (4)
In order to better directly model long-range interactions, we make the global
mapping f(;) recurrent, which leads to the following form:
rt=f(gt;rt1) (5)
where rtrepresents the recurrent object reasoning state at time tandgtis the

Object Level Visual Reasoning in Videos


rt=f(gt;rt1) (5)
where rtrepresents the recurrent object reasoning state at time tandgtis the
global inter-frame interaction inferred at time tsuch as described in Equation 4.
In practice, this is implemented as a GRU, but for simplicity we omitted the
gates in Equation (5). The pairwise mappings h(;) are implemented as an
MLP. Figure 3 provides a visual explanation of the object head's operating
through time.
Our proposed ORN diers from [32] in three main points:
Objects have a semantic denition | we model relationships with respect
to semantically meaningful entities (object instances) instead of feature map
cells which do not have a semantically meaningful spatial extent. We will show
in the experimental section that this is a key dierence.
Object Level Visual Reasoning in Videos 9
Objects are selected from dierent frames | we infer object pairwise
relations only between objects present in two dierent sets. This is a key design
choice which allows our model to reason about changes in object relationships
over time.
Long range reasoning | integration of the object relations over time is recurrent by using a RNN for f(). Since reasoning from a full sequence cannot

Object Level Visual Reasoning in Videos


over time.
Long range reasoning | integration of the object relations over time is recurrent by using a RNN for f(). Since reasoning from a full sequence cannot
be done by inferring the relations between two frames, f() allows long range
reasoning on sequences of variable length.
3.2 Object instance features
The object features Ot=
ok
t	K
k=1for each frame tused for the ORN module
described above are computed and collected from local regions predicted by
a mask predictor. Independently for each frame Xtof the input data block,
we predict object instances as binary masks Bk
tand associated object class
predictions ck
t, a distribution over Cclasses. We use Mask-RCNN [15], which
is able to detect objects in a frame using region proposal networks [30] and
produces a high quality segmentation mask for each object instance.
The objective is to collect features for each object instance, which jointly
describe its appearance, the change in its appearance over time, and its shape,
i.e. the shape of the binary mask. In theory, appearance could also be described
by pooling the feature representation learned by the mask predictor (Mask RCNN). However, in practice we choose to pool features from the dedicated object

Object Level Visual Reasoning in Videos


by pooling the feature representation learned by the mask predictor (Mask RCNN). However, in practice we choose to pool features from the dedicated object
head such as shown in Figure 2, which also include motion through the spatiotemporal convolutions shared with the activity head:
uk
t= ROI-Pooling( Ut;Bk
t) (6)
where Utis the feature map output by the object head ,uk
tis aD-dimensional
vector of appearance and appearance change of object k.
Shape information from the binary mask Bk
tis extracted through the following mapping function: bk
t=g(Bk
t);where g() is a MLP. Information about
object kin image Xtis given by a concatenation of appearance, shape, and
object class: ok
t= [bk
tuk
tck
t].
3.3 Global Motion and Context
Current approaches in video understanding focus on modeling the video from
a high-level perspective. By a stack of spatio-temporal convolution and pooling
they focus on learning global scene context information. Eective activity recognition requires integration of both of these sources: global information about the
entire video content in addition to relational reasoning for making ne distinctions regarding object interactions and properties.
In our method, local low-level reasoning is provided through object head

Object Level Visual Reasoning in Videos


entire video content in addition to relational reasoning for making ne distinctions regarding object interactions and properties.
In our method, local low-level reasoning is provided through object head
and the ORN module such as described above in Section 3.1. We complement
10 Baradel et al.
this representation by high-level context information described by Vtwhich are
feature outputs from the activity head (orange block in Figure 2).
We use spatial global average pooling over Vtto output T D-dimensional
feature vectors denoted by vt, where vtcorresponds to the context information
of the video at timestep t.
We model the dynamics of the context information through time by employing a RNN f
() given by:
st=f
(vt;st1) (7)
where sis the hidden state of f
() and gives cues about the evolution of the
context though time.
3.4 Recognition
Given an input video sequence X1:T, the two dierent streams corresponding to
the activity head and the object head result in the two representations handr,
respectively where h=P
thtandr=P
trt. Each representation is the hidden
state of the respective GRU, which were described in the preceding subsections.
Recall that hprovides the global motion context while rprovides the object

Object Level Visual Reasoning in Videos


state of the respective GRU, which were described in the preceding subsections.
Recall that hprovides the global motion context while rprovides the object
reasoning state output by the ORN module. We perform independent linear
classication for each representation:
y1=W h (8)
y2=Z r (9)
where y1;y2correspond to the logits from the activity head and the object head ,
respectively, and WandZare trainable weights (including biases). The nal
prediction is done by averaging logits y1andy2followed by softmax activation.
4 Network Architectures and feature dimensions
The input RGB images Xtare of size R3WHwhere WandHcorrespond to
the width and height and are of size 224 each. The object and activity heads
(orange and green in Figure 2) are a joint convolutional neural network with
Resnet50 architecture pre-trained on ImageNet/ILSVRC [31], with Conv1 and
Conv5 blocks being in
ated to 2.5D convolutions [44] (3D convolutions with a
separable temporal dimension). This choice has been optimized on the validation
set, as explained in Section 6 and shown in Table 5.
The last conv5 layers have been split into two dierent heads (activity head

Object Level Visual Reasoning in Videos


set, as explained in Section 6 and shown in Table 5.
The last conv5 layers have been split into two dierent heads (activity head
and object head). The intermediate feature representations UtandVtare of dimensions 2048T1414 and 2048T77, respectively. We provide a higher
spatial resolution for the feature maps Utof the object head to get more precise
local descriptors. This can be done by changing the stride of the initial conv5
layers from 2 to 1. Temporal convolutions have been congured to keep the same
time temporal dimension through the network.
Object Level Visual Reasoning in Videos 11
Global spatial pooling of activity features results in a 2048 dimensional feature vector fed into a GRU with 512 dimensional hidden state st. ROI-Pooling
of object features results in 2048 dimensional feature vectors uk
t. The encoder of
the binary mask is a MLP with one hidden layer of size 100 and outputs a mask
embedding bk
tof dimension 100. The number of object classes is 80, which leads
in total to a 2229 dimensional object feature vector ok
t.
The non-linearity h() is implemented as an MLP with 2 hidden layers each

Object Level Visual Reasoning in Videos


in total to a 2229 dimensional object feature vector ok
t.
The non-linearity h() is implemented as an MLP with 2 hidden layers each
with 512 units and produces an 512 dimensional output space. f() is implemented as a GRU with a 256 dimension hidden state rt. We use ReLU as the
activation function after each layer for each network.
5 Training
We train the model with a loss split into two terms:
Ltot=L^y1+^y2
2;y
+X
tX
kL(^ck
t;ck
t): (10)
whereLis the cross-entropy loss. The rst term corresponds to supervised activity class losses comparing two dierent activity class predictions to the class
ground truth: ^y1is the prediction of the activity head, whereas ^y2is the prediction of the object head, as given by Equations (8) and (9), respectively.
The second term is a loss which pushes the features Uof the object towards
representations of the semantic object classes. The goal is to obtain features
related to, both, motion (through the layers shared with the activity head), as
well as as object classes. As ground-truth object classes are not available, we
dene the loss as the cross-entropy between the class label ck

Object Level Visual Reasoning in Videos


well as as object classes. As ground-truth object classes are not available, we
dene the loss as the cross-entropy between the class label ck
tpredicted by the
mask predictor and a dedicated linear class prediction ^ck
tbased on features uk
t,
which, as we recall, are RoI-pooled from U:
ck
t=R uk
t (11)
where Rtrainable parameters (biases integrated) learned end-to-end together
with the other parameters of the model.
We found that rst training the object head only and then the full network
was performing better. A ResNet50 network pretrained on ImageNet is modied
by in
ating some of its lters to 2.5 convolutions (3D convolutions with the time
dimension separated), as described in Section 4; then by ne-tuning.
We train the model using the Adam optimizer [22] with an initial learning
rate of 104on 30 epochs and use early-stopping criterion on the validation set
for hyper-parameter optimization. Training takes 50 minutes per epoch on 4
Titan XP GPUs with clips of 8 frames.
12 Baradel et al.
6 Experimental results
We evaluated the method on three standard datasets, which represent dicult

Object Level Visual Reasoning in Videos


Titan XP GPUs with clips of 8 frames.
12 Baradel et al.
6 Experimental results
We evaluated the method on three standard datasets, which represent dicult
ne-grained activity recognition tasks: the Something-Something dataset, the
VLOG dataset and the recently released EPIC Kitchens dataset.
Something-Something (SS) is a recent video classication dataset with
108,000 example videos and 157 classes [13]. It shows humans performing dierent actions with dierent objects, actions and objects being combined in dierent
ways. Solving SS requires common sense reasoning and the state-of-the-art methods in activity recognition tend to fail, which makes this dataset challenging.
VLOG is a multi-label binary classication of human-object interactions
recently released with 114,000 videos and 30 classes [12]. Classes correspond
to objects, and labels of a class are 1 if a person has touched a certain object
during the video, otherwise they are 0. It has recently been shown, that stateof-the-art video based methods [6] are outperformed on VLOG by image based
methods like ResNet-50 [16], although these video methods outperform image
based ResNet-50 on large-scale video datasets like the Kinetics dataset [6]. This

Object Level Visual Reasoning in Videos


methods like ResNet-50 [16], although these video methods outperform image
based ResNet-50 on large-scale video datasets like the Kinetics dataset [6]. This
suggests a gap between traditional datasets like Kinetics and the ne-grained
dataset VLOG, making it particularly dicult.
EPIC Kitchens (EPIC) is an egocentric video dataset recently released
containing 55 hours recording of daily activities [7]. This is the largest in rstperson vision and the activities performed are non-scripted, which makes the
dataset very challenging and close to real world data. The dataset is densely
annotated and several tasks exist such as object detection, action recognition and
action prediction. We focus on action recognition with 39'594 action segments
in total and 125 actions classes (i.e verbs). Since the test set is not available yet
we conducted our experiments on the training set (28'561 videos). We use the
videos recorded by person 01 to person 25 for training (22'675 videos) and dene
the validation set as the remaining videos (5'886 videos).
For all datasets we rescale the input video resolution to 256 256. While
training, we crop space-time blocks of 224 224 spatial resolution and Lframes,

Object Level Visual Reasoning in Videos


For all datasets we rescale the input video resolution to 256 256. While
training, we crop space-time blocks of 224 224 spatial resolution and Lframes,
withL=8 for the SS dataset and L=4 for VLOG and EPIC. We do not perform
any other data augmentation. While training we extract Lframes from the entire
video by splitting the video into Lsub-sequences and randomly sampling one
frame per sub-sequence. The output sequence of size Lis called a clip. A clip
aims to represent the full video with less frames. For testing we aggregate results
of 10 clips. We use lintel [10] for decoding video on the
y.
The ablation study is done by using the train set as training data and we
report the result on the validation set. We compare against other state-of-theart approaches on the test set. For the ablation studies, we slightly decreased
the computational complexity of the model: the base network (including activity
and object heads) is a ResNet-18 instead of ResNet-50, a single clip of 4 frames
is extracted from a video at test time.
Comparison with other approaches. Table 1 shows the performance of
the proposed approach on the VLOG dataset. We outperform the state of the art
Object Level Visual Reasoning in Videos 13

Object Level Visual Reasoning in Videos


Comparison with other approaches. Table 1 shows the performance of
the proposed approach on the VLOG dataset. We outperform the state of the art
Object Level Visual Reasoning in Videos 13
Table 1. Results on Hand/Semantic Object Interaction Classication (Average precision in % on the test set) on VLOG dataset. R50 and I3D implemented by [12].
mAP
bag
bed
bedding
book/papers
bottle/tube
bowl
box
brush
cabinet
cell-phone
clothing
cup
door
drawers
food
fork
knife
laptop
microwave
oven
pen/pencil
pillow
plate
refrigerator
sink
spoon
stued animal
table
toothbrush
towel
R50 [16] 40.5 29.7 68.9 65.8 64.5 58.2 33.1 22.1 19.0 23.9 54.0 45.5 28.6 49.2 28.7 49.6 19.4 37.5 62.9 48.8 23.0 36.9 39.2 12.5 55.9 58.8 31.1 57.4 26.8 39.6 22.9

Object Level Visual Reasoning in Videos


I3D [6] 39.7 24.9 71.7 71.4 62.5 57.1 27.1 19.2 33.9 20.7 50.6 45.8 24.7 54.7 19.1 50.8 19.3 41.9 54.0 27.5 21.4 37.4 42.9 12.6 42.5 60.4 33.9 46.0 23.5 59.6 34.7
Ours 44.7 30.2 72.3 70.7 64.9 59.8 38.2 24.6 26.3 22.4 64.5 47.2 35.4 57.9 25.2 48.5 24.5 40.2 72.0 54.1 26.5 39.9 48.6 15.2 53.5 60.7 36.8 52.8 27.9 64.0 37.6
on this challenging dataset by a margin of 4.2 points (44.7% accuracy against
40.5% by [16]). As mentioned above, traditional video approaches tend to fail
on this challenging ne-grained dataset, providing inferior results. Table 3 shows
performance on SS where we outperform the state of the art given by very recent
methods (+2.3 points). On EPIC we re-implement standard baselines and report

Object Level Visual Reasoning in Videos


performance on SS where we outperform the state of the art given by very recent
methods (+2.3 points). On EPIC we re-implement standard baselines and report
results on the validation set (Table 4) since the test set is not available. Our full
method reports an accuracy of 40.89 and outperforms baselines by a large margin
(+6.4 and+7.9 points respectively for against CNN-2D and I3D based on a
ResNet-18).
Eect of object-level reasoning. Table 2 shows the importance of reasoning on the performance of the method. The baseline corresponds to the performance obtained by the activity head trained alone (in
ated ResNet, in the
ResNet-18 version for this table). No object level reasoning is present in this
baseline. The proposed approach (third line) including an object head and the
ORN module gains 0.8, 2.5 and 2.4 points compared to our baseline respectively
on SS, on EPIC and on VLOG. This indicates that the reasoning module is able
to extract complementary features compared to the activity head.
Using semantically dened objects proved to be important and led to a gain of
2 points on EPIC and 2.3 points on VLOG for the full model (6/12.7 points using

Object Level Visual Reasoning in Videos


2 points on EPIC and 2.3 points on VLOG for the full model (6/12.7 points using
the object head only) compared to an extension of Santoro et al. [32] operating
on pixel level. This indicates importance of object level reasoning. The gain on
SS is smaller (0.7 point with the full model and 7.8 points with the object head
only) and can be explained by the dierence in spatial resolution of the videos.
Object detections and predictions of the binary masks are done using the initial
video resolution. The mean video resolution for VLOG is 660 1183 and for EPIC
is 640480 against 100157 for SS. Mask-RCNN has been trained on images of
resolution 800800 and thus performs best on higher resolutions. The quality of
the object detector is important for leveraging object level understanding then
for the rest of the ablation study we focus on EPIC and VLOG datasets.
The function fin Equation (5) is an important design choice in our model.
In our proposed model, fis recurrent over time to ensure that the ORN module
captures long range reasoning over time, as shown in Equation (5). Removing
the recurrence in this equation leads to an MLP instead of a (gated) RNN, as

Object Level Visual Reasoning in Videos


the recurrence in this equation leads to an MLP instead of a (gated) RNN, as
evaluated in row 4 of Table 2. Performance decreases by 1.1 point on VLOG
and 1.4 points on EPIC. The larger gap for EPIC compared to VLOG and
can arguably be explained by the fact that in SS actions cover the whole video,
14 Baradel et al.
Table 2. Ablation study with ResNet-18 backbone. Results in %: Top-1 accuracy for
EPIC and SS datasets, and mAP for VLOG dataset.
Method Object typeEPIC VLOG SS
obj. head 2 heads obj. head 2 heads obj. head 2 heads
Baseline - - 38.33 - 35.03 - 31.31
ORN pixel 23.71 38.83 14.40 35.18 2.51 31.43
ORN COCO 29.94 40.89 27.14 37.49 10.26 32.12

Object Level Visual Reasoning in Videos


ORN COCO 29.94 40.89 27.14 37.49 10.26 32.12
ORN-mlp COCO 28.15 39.41 25.40 36.35 - ORN COCO-visual 28.45 38.92 22.92 35.49 - ORN COCO-shape 21.92 37.16 7.18 35.39 - ORN COCO-class 21.96 37.75 13.40 35.94 - ORN COCO-intra 29.25 38.10 26.78 36.28 - ORN clique-1 COCO 28.25 40.18 26.48 36.71 - ORN clique-3 COCO 22.61 37.67 27.05 36.04 - Table 3. Experimental results on the
Something-Something dataset (classication accuracy in % on the test set).
Methods Top1
C3D + Avg [13] 21.50
I3D [13] 27.63
MultiScale TRN [5] 33.60
Ours 35.97Table 4. Experimental results on the
EPIC Kitchens dataset (accuracy in %
on the validation set { methods with
have been re-implemented).
Methods Top1
R18 [16]32.05
I3D-18 [6]34.20

Object Level Visual Reasoning in Videos


have been re-implemented).
Methods Top1
R18 [16]32.05
I3D-18 [6]34.20
Ours 40.89
while solving VLOG requires detecting the right moment when the human-object
interaction occurs and thus long range reasoning plays a less important role.
Visual features extracted from object regions are the most discriminative,
however object shapes and labels also provide complementary information. Finally, the last part of Table 2 evaluates the eect of the cliques size for modeling the interactions between objects and show that pairwise cliques outperform
cliques of size 1 and 3. We would like to recall, that even with unary cliques
only, interactions between objects are still modeled. However, the model needs
to nd subspaces in the hidden representations associated to each interaction.
CNN architecture and kernel in
ations. The convolutional architecture
of the model was optimized over the validation set of the SS dataset, as shown
in Table 5. The architecture itself (in terms of numbers of layers, lters etc.)
is determined by pre-training on image classication. We optimized the choice
of lter in
ations from 2D to 2.5D or 3D for several convolutional blocks. This
has been optimized for the single head model and using a ResNet-18 variant

Object Level Visual Reasoning in Videos


has been optimized for the single head model and using a ResNet-18 variant
to speed up computation. Adding temporal convolutions increases performance
up to 100% w.r.t. to pure 2D baselines. This indicates, without surprise, that
motion is a strong cue. In
ating kernels to 2.5D on the input side and on the
Object Level Visual Reasoning in Videos 15
Table 5. Eect of the CNN architecture (choice of kernel in
ations) on a single head
ResNet-18 network. Accuracy in % on the validation set of Something-Something is
shown. 2.5D kernels are separable kernels: 2D followed by a 1D temporal.
Conv1 Conv2 Conv3 Conv4 Conv5 AggregSS2D 3D 2.5D 2D 3D 2.5D 2D 3D 2.5D 2D 3D 2.5D 2D 3D 2.5D GAP RNN
X- - X- - X- - X- - X- - X -15.73
X- - X- - X- - X- - X- - -X 15.88
-X - -X - -X - -X - -X -X -31.42
- - X - - X - - X - - X - - XX -27.58

Object Level Visual Reasoning in Videos


-X - -X - -X - -X - -X -X -31.42
- - X - - X - - X - - X - - XX -27.58
X- - X- - X- - X- - -X -X -31.28
X- - X- - X- - -X - -X -X -32.06
X- - X- - -X - -X - -X -X -32.25
X- - X- - X- - X- - - - XX -31.31
X- - X- - X- - - - X - - XX -32.79
X- - X- - - - X - - X - - XX -33.77
-X -X- - X- - X- - X- - X -28.71
-X - -X -X- - X- - X- - X -31.42
- - XX- - X- - X- - X- - X -20.05
- - X - - XX- - X- - X- - X -22.52
output side provided best performances, suggesting that temporal integration
is required at a very low level (motion estimation) as well as on a very high
level, close to reasoning. Our study also corroborates recent research in activity

Object Level Visual Reasoning in Videos


is required at a very low level (motion estimation) as well as on a very high
level, close to reasoning. Our study also corroborates recent research in activity
recognition, indicating that 2.5D kernels provide a good trade-o between highcapacity and learnable numbers of parameters. Finally temporal integration via
RNN outperforms global average pooling over space and time. The choice of
a (gated) RNN for temporal integration of the activity head features proved
important (see Table 5) compared to global average pooling (GAP) over space
and time.
Visualizing the learned object interactions. Figure 4 shows visualizations of the pairwise object relationships the model learned from data, in particular from the VLOG dataset. Each graph is computed for a given activity class,
and strong arcs between two nodes in the graph indicate strong relationships
between the object classes, i.e. the model detects a high correlation between
these relationships and the activity. The graphs were obtained by thresholding
the summed activations of each pairwise relationship ( j; k) in equation (4). Each
pair ( j; k) can be assigned a pair of object classes cj
tandck
tthrough the predictions of the object instance mask predictor. Integrating over all samples of the
dataset for a given class leads to the visualizations in Figure 4. We can see that

Object Level Visual Reasoning in Videos


tthrough the predictions of the object instance mask predictor. Integrating over all samples of the
dataset for a given class leads to the visualizations in Figure 4. We can see that
the object interactions are highly relevant to the detected activities: the persontouches-bed activity is correlated to interactions between relevant object classes
person and bed. Similarly, activities human-bowl interaction and human-cup interaction show interactions with the respective objects bowl and cup. Moreover,
other recovered relationships are highly correlated to the scene (for example,
dining-table and bowl for activity human-bowl interaction ).
16 Baradel et al.
Fig. 4. Example of object pairwise interactions learned by our model on VLOG for
four dierent classes. Objects co-occurrences are at the top and learned pairwise objects
interactions are at the bottom. Line thickness indicates learned importance of a given
relation. Interactions have been normalized by the object co-occurrences.
teddy bearteddy bearteddy bearbedcell phonepersonpersonbedteddy bear
bottlewine glasspersonpersonpersonpersonbottle
Fig. 5. Examples of failure cases { a) small sized objects (on the left). Our model
detects a cell phone and a person but fails to detect hand-cell-phone contact ; b) confusion between semantically similar objects (on the right). The model falsly predicts

Object Level Visual Reasoning in Videos


detects a cell phone and a person but fails to detect hand-cell-phone contact ; b) confusion between semantically similar objects (on the right). The model falsly predicts
hand-cup contact instead of hand-glass-contact even though the wine glass is detected.
Finally, Figure 5 shows some failure cases, which are either due to errors
done by the object mask prediction (Mask R-CNN) or by the ORN itself.
7 Conclusion
We presented a method for activity recognition in videos which leverages object instance detections for visual reasoning on object interactions over time.
The choice of reasoning over semantically well-dened objects is key to our approach and outperforms state of the art methods which reason on grid-levels,
such as cells of convolutional feature maps. Temporal dependencies and causal
relationships are dealt with by integrating relationships between dierent time
instants. We evaluated the method on three dicult datasets, on which standard
approaches do not perform well, and report state-of-the-art results.
Acknowledgements. This work was funded by grant Deepvision (ANR-15CE23-0029, STPGP-479356-15), a joint French/Canadian call by ANR & NSERC.
Object Level Visual Reasoning in Videos 17
References

Object Level Visual Reasoning in Videos


Object Level Visual Reasoning in Videos 17
References
1. Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B.,
Vijayanarasimhan, S.: Youtube-8m: A large-scale video classication benchmark.
arXiv preprint arxiv:1609.08675 (2016)
2. Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.: Sequential deep
learning for human action recognition. In: HBU (2011)
3. Baradel, F., Wolf, C., Mille, J., Taylor, G.: Glimpse clouds: Human activity recognition from unstructured feature points. In: CVPR (2018)
4. Battaglia, P.W., Pascanu, R., Lai, M., Rezende, D.J., Kavukcuoglu, K.: Interaction
networks for learning about objects, relations and physics. In: NIPS (2016)
5. Bolei, Z., Zhang, A.A., Torralba, A.: Temporal relational reasoning in videos. In:
ECCV (2018)

Object Level Visual Reasoning in Videos


5. Bolei, Z., Zhang, A.A., Torralba, A.: Temporal relational reasoning in videos. In:
ECCV (2018)
6. Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the
kinetics dataset. In: CVPR (2017)
7. Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E.,
Moltisanti, D., Munro, J., Perrett, T., Price, W., Wray, M.: Scaling egocentric
vision: The epic-kitchens dataset. In: ECCV (2018)
8. Deng, Z., Vahdat, A., Hu, H., Mori, G.: Structure inference machines: Recurrent
neural networks for analyzing relations in group activity recognition. In: CVPR
(2016)
9. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S.,
Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual
recognition and description. In: CVPR (2015)

Object Level Visual Reasoning in Videos


Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual
recognition and description. In: CVPR (2015)
10. Duke, B.: Lintel: Python video decoding. https://github.com/dukebw/lintel
(2018)
11. Fleuret, F., Li, T., Dubout, C., Wampler, E.K., Yantis, S., Geman, D.: Comparing
machines and humans on a visual categorization test. Proceedings of the National
Academy of Sciences of the United States of America 108 43 , 17621{5 (2011)
12. Fouhey, D.F., Kuo, W., Efros, A.A., Malik, J.: From lifestyle vlogs to everyday
interactions. In: CVPR (2018)
13. Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim,
H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., Hoppe, F., Thurau,
C., Bax, I., Memisevic, R.: The "something something" video database for learning
and evaluating visual common sense. In: ICCV (2017)

Object Level Visual Reasoning in Videos


C., Bax, I., Memisevic, R.: The "something something" video database for learning
and evaluating visual common sense. In: ICCV (2017)
14. Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan,
S., Toderici, G., Ricco, S., Sukthankar, R., Schmid, C., Malik, J.: Ava: A
video dataset of spatio-temporally localized atomic visual actions. arXiv preprint
arXiv:1705.08421 (2017)
15. He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask r-cnn. In: ICCV (2017)
16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.
In: CVPR (2016)
17. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation
9(8), 1735{1780 (1997)
18. Hudson, D., Manning, C.: Compositional attention networks for machine reasoning.
In: ICLR (2018)

Object Level Visual Reasoning in Videos


9(8), 1735{1780 (1997)
18. Hudson, D., Manning, C.: Compositional attention networks for machine reasoning.
In: ICLR (2018)
19. Jain, A., Zamir, A.R., Savarese, S., Saxena, A.: Structural-RNN: Deep Learning
on Spatio-Temporal Graphs. In: CVPR (2016)
20. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Largescale video classication with convolutional neural networks. In: CVPR (2014)
18 Baradel et al.
21. Kim, J., Ricci, M., Serre, T.: Not-so-CLEVR: Visual relations strain feedforward
neural networks. arXiv preprint arXiv:1802.03390 (2018)
22. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: ICML (2015)

Object Level Visual Reasoning in Videos


22. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: ICML (2015)
23. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalanditis, Y., Li, L.J., Shamma, D.A., Bernstein, M., Fei-Fei, L.: Visual genome:
Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV) 123, 32{73 (2017)
24. Luc, P., , Neverova, N., Couprie, C., Verbeek, J., LeCun, Y.: Predicting deeper
into the future of semantic segmentation. In: ICCV (2017)
25. Luvizon, D., Picard, D., Tabia, H.: 2d/3d pose estimation and action recognition
using multitask deep learning. In: CVPR (2018)
26. Monfort, M., Zhou, B., Bargal, S.A., Andonian, A., Yan, T., Ramakrishnan,
K., Brown, L., Fan, Q., Gutfruend, D., Vondrick, C., Oliva, A.: Moments
in time dataset: one million videos for event understanding. arXiv preprint

Object Level Visual Reasoning in Videos


in time dataset: one million videos for event understanding. arXiv preprint
arXiv:1801.03150 (2018)
27. Perez, E., Vries, H.D., Strub, F., Dumoulin, V., Courville, A.: Learning visual reasoning without strong priors. In: ICML Machine Learning in Speech and Language
Processing Workshop (2017)
28. Pickup, L.C., Pan, Z., Wei, D., Shih, Y., Zhang, C., Zisserman, A., Scholkopf, B.,
Freeman, W.T.: Seeing the arrow of time. In: CVPR (2014)
29. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for
3d classication and segmentation. In: CVPR (2017)
30. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object
detection with region proposal networks. In: NIPS (2015)
31. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,

Object Level Visual Reasoning in Videos


31. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,
Karpathy, A., Khosla, A., Bernstein, M., Berg, A., Fei-Fei, L.: Imagenet large scale
visual recognition challenge. IJCV 115(3), 211{252 (2015)
32. Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia,
P., Lillicrap, T.: A simple neural network module for relational reasoning. In: NIPS
(2017)
33. Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+D: A Large Scale Dataset
for 3D Human Activity Analysis. In: CVPR (2016)
34. Sharma, S., Kiros, R., Salakhutdinov, R.: Action recognition using visual attention.
In: ICLR Workshop (2016)
35. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS (2014)

Object Level Visual Reasoning in Videos


35. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS (2014)
36. Song, S., Lan, C., Xing, J., Zeng, W., Liu, J.: An End-to-End Spatio-Temporal
Attention Model for Human Action Recognition from Skeleton Data. In: AAAI
(2016)
37. Stabinger, S., Rodr guez-S anchez, A., Piater, J.: 25 years of CNNs: Can we compare
to human abstraction capabilities? In: ICANN (2016)
38. van Steenkiste, S., Chang, M., Gre, K., Schmidhuber, J.: Relational neural expectation maximization: Unsupervised discovery of objects and their interactions.
In: ICLR (2018)
39. Sun, L., Jia, K., Chen, K., Yeung, D., Shi, B.E., Savarese, S.: Lattice long shortterm memory for human action recognition. In: ICCV (2017)
40. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: ICCV (2015)

Object Level Visual Reasoning in Videos


41. Velikovi, P., Cucurull, G., Casanova, A., Romero, A., Li, P., Bengio, Y.: Graph
attention networks. In: ICLR (2018)
Object Level Visual Reasoning in Videos 19
42. Wang, H., Kl aser, A., Schmid, C., Liu, C.L.: Action Recognition by Dense Trajectories. In: CVPR (2011)
43. Watters, N., Zoran, D., Weber, T., Battaglia, P., Pascanu, R., Tacchetti, A.: Visual
interaction networks: Learning a physics simulator from video. In: NIPS (2017)
44. Xie, S., Sun, C., Huang, J., Tu, Z., Murphy, K.: Rethinking spatiotemporal feature
learning for video understanding. arXiv prepring arxiv:1712.04851 (2017)
45. Yeung, S., Russakovsky, O., Jin, N., Andriluka, M., Mori, G., Fei-Fei, L.: Every moment counts: Dense detailed labeling of actions in complex videos. arXiv preprint
arXiv:1507.05738 (2015)

Object Level Visual Reasoning in Videos


arXiv:1507.05738 (2015)
46. Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola,
A.J.: Deep sets. In: NIPS. pp. 3391{3401 (2017), http://papers.nips.cc/paper/
6931-deep-sets.pdf

Object Level Visual Reasoning in Videos


Massively Multilingual Neural Machine Translation
Roee Aharoni
Bar Ilan University
Ramat-Gan
Israel
roee.aharoni@gmail.comMelvin Johnson andOrhan Firat
Google AI
Mountain View
California
melvinp,orhanf@google.com
Abstract
Multilingual neural machine translation
(NMT) enables training a single model that
supports translation from multiple source languages into multiple target languages. In this
paper, we push the limits of multilingual NMT
in terms of the number of languages being
used. We perform extensive experiments in
training massively multilingual NMT models,
translating up to 102 languages to and from
English within a single model. We explore
different setups for training such models and
analyze the trade-offs between translation
quality and various modeling decisions. We
report results on the publicly available TED
talks multilingual corpus where we show
that massively multilingual many-to-many
models are effective in low resource settings,
outperforming the previous state-of-the-art
while supporting up to 59 languages. Our
experiments on a large-scale dataset with
102 languages to and from English and up to
one million examples per direction also show
promising results, surpassing strong bilingual
baselines and encouraging future work on
massively multilingual NMT.
1 Introduction

Summary:
Multilingual neural machine translation (NMT) enables training a single model
that supports translation from multiple source languages into multiple target
languages. In this paper, we push the limits of multilingual NMT in terms of
number of languages being used. We perform extensive experiments in training
massively multilingual NMT models, translating up to 102 languages to and from
English within a single model. We explore different setups for training such
models and analyze the trade-offs between translation quality and various
modeling decisions. We report results on the publicly available TED talks
multilingual corpus where we show that massively multilingual many-to-many
models are effective in low resource settings, outperforming the previous
state-of-the-art while supporting up to 59 languages. Our experiments on a
large-scale dataset with 102 languages to and from English and up to one
million examples per direction also show promising results, surpassing strong
bilingual baselines and encouraging future work on massively multilingual NMT.

Accepted as a long paper in NAACL 2019


one million examples per direction also show
promising results, surpassing strong bilingual
baselines and encouraging future work on
massively multilingual NMT.
1 Introduction
Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2014;
Sutskever et al., 2014) is the current state-ofthe-art approach for machine translation in both
academia (Bojar et al., 2016, 2017, 2018) and industry (Wu et al., 2016; Hassan et al., 2018). Recent works (Dong et al., 2015; Firat et al., 2016a;
Ha et al., 2016; Johnson et al., 2017) extended the
approach to support multilingual translation, i.e.
training a single model that is capable of translating between multiple language pairs.
Multilingual models are appealing for several
reasons. First, they are more efﬁcient in terms
Work carried out during an internship at Google AI.of the number of required models and model parameters, enabling simpler deployment. Another
beneﬁt is transfer learning; when low-resource
language pairs are trained together with highresource ones, the translation quality may improve
(Zoph et al., 2016; Nguyen and Chiang, 2017). An
extreme case of such transfer learning is zero-shot

Accepted as a long paper in NAACL 2019


(Zoph et al., 2016; Nguyen and Chiang, 2017). An
extreme case of such transfer learning is zero-shot
translation (Johnson et al., 2017), where multilingual models are able to translate between language
pairs that were never seen during training.
While very promising, it is still unclear how far
one can scale multilingual NMT in terms of the
number of languages involved. Previous works
on multilingual NMT typically trained models
with up to 7 languages (Dong et al., 2015; Firat et al., 2016b; Ha et al., 2016; Johnson et al.,
2017; Gu et al., 2018) and up to 20 trained directions (Cettolo et al., 2017) simultaneously. One
recent exception is Neubig and Hu (2018) who
trained many-to-one models from 58 languages
into English. While utilizing signiﬁcantly more
languages than previous works, their experiments
were restricted to many-to-one models in a lowresource setting with up to 214k examples per
language-pair and were evaluated only on four
translation directions.
In this work, we take a step towards practical
“universal” NMT – training massively multilingual models which support up to 102 languages
and with up to one million examples per languagepair simultaneously. Speciﬁcally, we focus on

Accepted as a long paper in NAACL 2019


and with up to one million examples per languagepair simultaneously. Speciﬁcally, we focus on
training “English-centric” many-to-many models,
in which the training data is composed of many
language pairs that contain English either on the
source side or the target side. This is a realistic
setting since English parallel data is widely available for many language pairs. We restrict our experiments to Transformer models (Vaswani et al.,
2017) as they were shown to be very effective
in recent benchmarks (Ott et al., 2018), also inarXiv:1903.00089v3  [cs.CL]  2 Jul 2019
the context of multilingual models (Lakew et al.,
2018; Sachan and Neubig, 2018).
We evaluate the performance of such massively
multilingual models while varying factors like
model capacity, the number of trained directions
(tasks) and low-resource vs. high-resource settings. Our experiments on the publicly available
TED talks dataset (Qi et al., 2018) show that massively multilingual many-to-many models with up
to 58 languages to-and-from English are very effective in low resource settings, allowing to use
high-capacity models while avoiding overﬁtting

Accepted as a long paper in NAACL 2019


to 58 languages to-and-from English are very effective in low resource settings, allowing to use
high-capacity models while avoiding overﬁtting
and achieving superior results to the current stateof-the-art on this dataset (Neubig and Hu, 2018;
Wang et al., 2019) when translating into English.
We then turn to experiment with models trained
on 103 languages in a high-resource setting. For
this purpose we compile an English-centric inhouse dataset, including 102 languages aligned
to-and-from English with up to one million examples per language pair. We then train a single model on the resulting 204 translation directions and ﬁnd that such models outperform strong
bilingual baselines by more than 2 BLEU averaged across 10 diverse language pairs, both toand-from English. Finally, we analyze the tradeoffs between the number of involved languages
and translation accuracy in such settings, showing
that massively multilingual models generalize better to zero-shot scenarios. We hope these results
will encourage future research on massively multilingual NMT.
2 Low-Resource Setting: 59 Languages
2.1 Experimental Setup
The main question we wish to answer in this work
is how well a single NMT model can scale to
support a very large number of language pairs.
The answer is not trivial: on the one hand, training multiple language pairs together may result

Accepted as a long paper in NAACL 2019


is how well a single NMT model can scale to
support a very large number of language pairs.
The answer is not trivial: on the one hand, training multiple language pairs together may result
in transfer learning (Zoph et al., 2016; Nguyen
and Chiang, 2017). This may improve performance as we increase the number of language
pairs, since more information can be shared between the different translation tasks, allowing the
model to learn which information to share. On the
other hand, adding many language pairs may result in a bottleneck; the model has a limited capacity while it needs to handle this large number
of translation tasks, and sharing all parameters between the different languages can be sub-optimal(Wang et al., 2018) especially if they are not from
the same typological language family (Sachan and
Neubig, 2018).
We begin tackling this question by experimenting with the TED Talks parallel corpus compiled
by Qi et al. (2018)1, which is unique in that it includes parallel data from 59 languages. For comparison, this is signiﬁcantly “more multilingual”
than the data available from all previous WMT
news translation shared task evaluations throughout the years – the latest being Bojar et al. (2016,
2017, 2018), which included 14 languages so far.2
We focus on the setting where we train

Accepted as a long paper in NAACL 2019


2017, 2018), which included 14 languages so far.2
We focus on the setting where we train
“English-centric” models, i.e. training on all
language pairs that contain English in either the
source or the target, resulting in 116 translation
directions. This dataset is also highly imbalanced, with language pairs including between 3.3k
to 214k sentence pairs for training. Table 9 in
the supplementary material details the languages
and training set sizes for this dataset. Since the
dataset is already tokenized we did not apply additional preprocessing other than applying joint
subword segmentation (Sennrich et al., 2016) with
32k symbols.
Regarding the languages we evaluate on, we begin with the same four languages as Neubig and
Hu (2018) – Azerbeijani (Az), Belarusian (Be),
Galician (Gl) and Slovak (Sk). These languages
present an extreme low-resource case, with as few
as 4.5k training examples for Belarusian-English.
In order to better understand the effect of training
set size in these settings, we evaluate on four additional languages that have more than 167k training examples each – Arabic (Ar), German (De),
Hebrew (He) and Italian (It).
2.2 Model Details
Using the same data, we trained three massively

Accepted as a long paper in NAACL 2019


Hebrew (He) and Italian (It).
2.2 Model Details
Using the same data, we trained three massively
multilingual models: a many-to-many model
which we train using all 116 translation directions
with 58 languages to-and-from English, a one-tomany model from English into 58 languages, and
a many-to-one model from 58 languages into English. We follow the method of Ha et al. (2016);
Johnson et al. (2017) and add a target-language
1github.com/neulab/
word-embeddings-for-nmt
2Chinese, Czech, English, Estonian, Finnish, French,
German, Hindi, Hungarian, Latvian, Romanian, Russian,
Spanish, Turkish. According to http://www.statmt.
org/wmtXX
preﬁx token to each source sentence to enable
many-to-many translation. These different setups
enable us to examine the effect of the number of
translation tasks on the translation quality as measured in BLEU (Papineni et al., 2002). We also
compare our massively multilingual models to
bilingual baselines and to two recently published
results on this dataset (Neubig and Hu (2018);
Wang et al. (2019)).

Accepted as a long paper in NAACL 2019


bilingual baselines and to two recently published
results on this dataset (Neubig and Hu (2018);
Wang et al. (2019)).
Regarding the models, we focused on the Transformer in the “Base” conﬁguration. We refer the
reader to Vaswani et al. (2017) for more details
on the model architecture. Speciﬁcally, we use 6
layers in both the encoder and the decoder, with
model dimension set at 512, hidden dimension
size of 2048 and 8 attention heads. We also applied dropout at a rate of 0.2 in the following components: on the sum of the input embeddings and
the positional embeddings, on the output of each
sub-layer before added to the previous layer input
(residual connection), on the inner layer output after the ReLU activation in each feed-forward sublayer, and to the attention weight in each attention
sub-layer. This results in a model with approximately 93M trainable parameters. For all models we used the inverse square root learning rate
schedule from Vaswani et al. (2017) with learningrate set at 3 and 40k warmup steps. All models are
implemented in Tensorﬂow-Lingvo (Shen et al.,
2019).

Accepted as a long paper in NAACL 2019


implemented in Tensorﬂow-Lingvo (Shen et al.,
2019).
In all cases we report test results for the checkpoint that performed best on the development set
in terms of BLEU. For the multilingual models we
create a development set that includes examples
we uniformly sample from a concatenation of all
the individual language pair development sets, resulting in 13k development examples per model.
Another important detail regarding multilingual
training is the batching scheme. In all of our multilingual models we use heterogeneous batching,
where each batch contains examples which are
uniformly sampled from a concatenation of all the
language pairs the model is trained on. Speciﬁcally, we use batches of 64 examples for sequences
shorter than 69 tokens and batches of 16 examples for longer sequences. We did not use oversampling as the dataset is relatively small.
2.3 Results
We use tokenized BLEU in order to be comparable with Neubig and Hu (2018). Table 1 showsAz-En Be-En Gl-En Sk-En Avg.
# of examples 5.9k 4.5k 10k 61k 20.3k
Neubig & Hu 18
baselines 2.7 2.8 16.2 24 11.42

Accepted as a long paper in NAACL 2019


Neubig & Hu 18
baselines 2.7 2.8 16.2 24 11.42
many-to-one 11.7 18.3 29.1 28.3 21.85
Wang et al. 18 11.82 18.71 30.3 28.77 22.4
Ours
many-to-one 11.24 18.28 28.63 26.78 21.23
many-to-many 12.78 21.73 30.65 29.54 23.67
Table 1: X !En test BLEU on the TED Talks corpus,
for the language pairs from Neubig and Hu (2018)
Ar-En De-En He-En It-En Avg.
# of examples 213k 167k 211k 203k 198.5k
baselines 27.84 30.5 34.37 33.64 31.59
many-to-one 25.93 28.87 30.19 32.42 29.35
many-to-many 28.32 32.97 33.18 35.14 32.4
Table 2: X !En test BLEU on the TED Talks corpus,
for language pairs with more than 167k examples
updates ar_many2one ro_many2one nl_many2one de_many2one it_many2one ar de it nl
0 0

Accepted as a long paper in NAACL 2019


Accepted as a long paper in NAACL 2019


95667 0.2852652967
97386 0.286067158
99069 0.285282582
101647 0.2873030901
103358 0.2812287509
105932 0.2841886282
107656 0.2752591372
109350 0.2852451503
111069 0.2802457809
113631 0.2839408815
115325 0.2698247731
117038 0.2862476707
119599 0.2838998437
121302 0.2805997431
123015 0.2826767564
Figure 1: Development BLEU on
fIt,Ro,Nl,De,Ar g!En vs. training BLEU for the
many-to-one and many-to-many models. Best viewed
in color.
the results of our experiments when evaluating
on the same language pairs as they did. The results under “Neubig & Hu 18” are their bilingual baselines and their best many-to-one models.
Their many-to-one models use similar-languageregularization, i.e. ﬁne-tuning a pre-trained manyto-one model with data from the language pair of
interest together with data from a language pair

Accepted as a long paper in NAACL 2019


interest together with data from a language pair
that has a typologically-similar source language
and more training data (i.e. Russian and Belarusian, Turkish and Azerbaijani). The results under
“Ours” are our many-to-one and many-to-many
models we trained identically in terms of model
architecture and hyper-parameters.
We ﬁrst note that our many-to-many model outperforms all other models when translating into
English, with 1.82 BLEU improvement (when averaged across the four language pairs) over the
best ﬁne-tuned many-to-one models of Neubig
and Hu (2018) and 2.44 BLEU improvement over
our many-to-one model when averaged across the
four low-resource language pairs (Table 1). This
is surprising as it uses the same X !En data,
model architecture and capacity as our many-toone model, while handling a heavier burden since
it also supports 58 additional translation tasks
(from English into58 languages). Our models also
outperform the more complex models of Wang
et al. (2019) which use ”Soft Decoupled Encoding” for the input tokens, while our models use a
simple subword segmentation.

Accepted as a long paper in NAACL 2019


et al. (2019) which use ”Soft Decoupled Encoding” for the input tokens, while our models use a
simple subword segmentation.
One possible explanation is that the many-toone model overﬁts the English side of the corpus
as it is multi-way-parallel: in such setting the English sentences are overlapping across the different language pairs, making it much easier for the
model to memorize the training set instead of generalizing (when enough capacity is available). On
the other hand, the many-to-many model is trained
on additional target languages other than English,
which can act as regularizers for the X !En tasks,
reducing such overﬁtting.
To further illustrate this, Figure 1 tracks the
BLEU scores on the individual development sets
during training for Italian (It), Romanian (Ro),
Dutch (Nl), German (De) and Arabic (Ar) into English (left), together with BLEU scores on a subset of the training set for each model. We can
see that while the many-to-one model degrades in
performance on the development set, the manyto-many model still improves. Note the large
gap in the many-to-one model between the training set BLEU and the development set BLEU,
which points on the generalization issue that is

Accepted as a long paper in NAACL 2019


gap in the many-to-one model between the training set BLEU and the development set BLEU,
which points on the generalization issue that is
not present in the many-to-many setting. We also
note that our many-to-one model is on average
0.75 BLEU behind the best many-to-one models
in Neubig and Hu (2018). We attribute this to the
fact that their models are ﬁne-tuned using similarlanguage-regularization while our model is not.
We ﬁnd an additional difference between the
results on the resource-scarce languages (Table 1) and the higher-resource languages (Table
2). Speciﬁcally, the bilingual baselines outperform the many-to-one models only in the higherresource setting. This makes sense as in the low-En-Az En-Be En-Gl En-Sk Avg.
# of examples 5.9k 4.5k 10k 61k 20.3k
baselines 2.16 2.47 3.26 5.8 3.42
one-to-many 5.06 10.72 26.59 24.52 16.72
many-to-many 3.9 7.24 23.78 21.83 14.19
En-Ar En-De En-He En-It Avg.

Accepted as a long paper in NAACL 2019


many-to-many 3.9 7.24 23.78 21.83 14.19
En-Ar En-De En-He En-It Avg.
# of examples 213k 167k 211k 203k 198.5k
baselines 12.95 23.31 23.66 30.33 22.56
one-to-many 16.67 30.54 27.62 35.89 27.68
many-to-many 14.25 27.95 24.16 33.26 24.9
Table 3: En !X test BLEU on the TED Talks corpus
resource setting the baselines have very few training examples to outperform the many-to-one models, while in the higher resource setting they have
access to more training data. This corroborates the
results of Gu et al. (2018) that showed the sensitivity of such models to similar low resource conditions and the improvements gained from using
many-to-one models (however with much fewer
language pairs).
Table 3 shows the results of our massively
multilingual models and bilingual baselines when
evaluated out-of-English. In this case we see an
opposite trend: the many-to-many model performs
worse than the one-to-many model by 2.53 BLEU
on average. While previous works (Wang et al.,

Accepted as a long paper in NAACL 2019


worse than the one-to-many model by 2.53 BLEU
on average. While previous works (Wang et al.,
2018; Sachan and Neubig, 2018) discuss the phenomena of quality degradation in English-to-many
settings, this shows that increasing the number of
source languages also causes additional degradation in a many-to-many model. This degradation
may be due to the English-centric setting: since
most of the translation directions the model is
trained on are into English, this leaves less capacity for the other target languages (while still performing better than the bilingual baselines on all 8
language pairs). We also note that in this case the
results are consistent among the higher and lower
resource pairs – the one-to-many model is better
than the many-to-many model, which outperforms
the bilingual baselines in all cases. This is unlike
the difference we saw in the X !En experiments
since here we do not have the multi-way-parallel
overﬁtting issue.
2.4 Discussion
From the above experiments we learn that NMT
models can scale to 59 languages in a lowresource, imbalanced, English-centric setting,
with the following observations: (1) massively
multilingual many-to-many models outperform

Accepted as a long paper in NAACL 2019


models can scale to 59 languages in a lowresource, imbalanced, English-centric setting,
with the following observations: (1) massively
multilingual many-to-many models outperform
many-to-one and bilingual models with similar capacity and identical training conditions when averaged over 8 language pairs into English. We
attribute this improvement over the many-to-one
models to the multiple target language pairs which
may act as regularizers, especially in this lowresource multi-way-parallel setting that is prone to
memorization. (2) many-to-many models are inferior in performance when going out-of-English
in comparison to a one-to-many model. We attribute this to English being over-represented in
the English-centric many-to-many setting, where
it appears as a target language in 58 out of 116
trained directions, which may harm the performance on the rest of the target languages as the
model capacity is limited.3
It is important to stress the fact that we compared the different models under identical training
conditions and did not perform extensive hyperparameter tuning for each setting separately. However, we believe that such tuning may improve
performance even further, as the diversity in each
training batch is very different between the different settings. For example, while the baseline
model batches include only one language in the

Accepted as a long paper in NAACL 2019


performance even further, as the diversity in each
training batch is very different between the different settings. For example, while the baseline
model batches include only one language in the
source and one language in the target, the manyto-many model includes 59 languages in each
side with a strong bias towards English. These
differences may require tailored hyper-parameter
choices for each settings (i.e. different batch sizes,
learning rate schedules, dropout rates etc.) which
would be interesting to explore in future work.
In the following experiments we investigate
whether these observations hold using (1) an even
larger set of languages, and (2) a much larger,
balanced training corpus that is not multi-wayparallel.
3 High-Resource Setting: 103 Languages
3.1 Experimental Setup
In this setting we scale the number of languages
and examples per language pair further when
training a single massively multilingual model.
Since we are not aware of a publicly available resource for this purpose, we construct an in-house
dataset. This dataset includes 102 language pairs
which we “mirror” to-and-from English, with up
to one million examples per language pair. This
results in 103 languages in total, and 204 translation directions which we train simultaneously.

Accepted as a long paper in NAACL 2019


to one million examples per language pair. This
results in 103 languages in total, and 204 translation directions which we train simultaneously.
3This issue may be alleviated by over-sampling the nonEnglish-target pairs, but we leave this for future work.More details about this dataset are available in Table 4, and Table 10 in the supplementary material
details all the languages in the dataset.4
Similarly to our previous experiments, we compare the massively multilingual models to bilingual baselines trained on the same data. We tokenize the data using an in-house tokenizer and then
apply joint subword segmentation to achieve an
open-vocabulary. In this setting we used a vocabulary of 64k subwords rather than 32k. Since the
dataset contains 24k unique characters, a 32k symbol vocabulary will consist of mostly characters,
thereby increasing the average sequence length.
Regarding the model, for these experiments we
use a larger Transformer model with 6 layers in
both the encoder and the decoder, model dimension set to 1024, hidden dimension size of 8192,
and 16 attention heads. This results in a model
with approximately 473.7M parameters.5Since
the model and data are much larger in this case,
we used a dropout rate of 0.1 for our multilingual
models and tuned it to 0.3 for our baseline models

Accepted as a long paper in NAACL 2019


the model and data are much larger in this case,
we used a dropout rate of 0.1 for our multilingual
models and tuned it to 0.3 for our baseline models
as it improved the translation quality on the development set.
We evaluate our models on 10 languages from
different typological families: Semitic – Arabic
(Ar), Hebrew (He), Romance – Galician (Gl),
Italian (It), Romanian (Ro), Germanic – German
(De), Dutch (Nl), Slavic – Belarusian (Be), Slovak (Sk) and Turkic – Azerbaijani (Az) and Turkish (Tr). We evaluate both to-and-from English,
where each language pair is trained on up to one
million examples. As in the previous experiment,
we report test results from the model that performed best in terms of BLEU on the development
set.
4The average number of examples per language pair is
940k, as for 13 out of the 102 pairs we had less than one
million examples available.
5This is larger than the Transformer “Big” conﬁguration,
which includes approximately 213M trained parameters.
# of language pairs 102
examples per pair
min 63,879
max 1,000,000
average 940,087
std. deviation 188,194

Accepted as a long paper in NAACL 2019


one-to-many 12.08 9.92 15.6 31.39 20.01 33 31.06 28.43 17.67 17.68 21.68
many-to-many 10.57 9.84 14.3 28.48 17.91 30.39 29.67 26.23 18.15 15.58 20.11
Table 6: En !X test BLEU on the 103-language corpus
3.2 Results
Table 5 describes the results when translating into
English. First, we can see that both multilingual
models perform better than the baselines in terms
of average BLEU. This shows that massively multilingual many-to-many models can work well in
realistic settings with millions of training examples, 102 languages and 204 jointly trained directions to-and-from English. Looking more closely,
we note several different behaviors in comparison
to the low-resource experiments on the TED Talks
corpus. First, the many-to-one model here performs better than the many-to-many model. This
shows that the previous result was indeed due to
the pathologies of the low-resource dataset; when
the training data is large enough and not multiway-parallel there is no overﬁtting in the many-toone model, and it outperforms the many-to-many
model in most cases while they are trained identically.

Accepted as a long paper in NAACL 2019


model in most cases while they are trained identically.
One particular outlier in this case is German-toEnglish, where the many-to-one model is 2 BLEU
points below the many-to-many model. We examine the BLEU score of this language pair on its
dedicated German-English development set during training in the many-to-one model and ﬁnd
that it highly ﬂuctuates. We then measure the
performance on the test set for this language pair
by choosing the best checkpoint on the dedicated
German-English development set (instead of on
the mixed multilingual development set) and ﬁnd
it to be 38.07, which is actually higher in 1 BLEU
than the best result of the many-to-many model.
This shows that while training many languages together, there is no “silver bullet”: some languages
may suffer from severe interference during training (i.e. a reduction of 3 BLEU in this case, from38.07 to 35.05) while other languages continue to
improve with more updates.
Table 6 describes the results when translating
out-of-English. Again, both of the massively multilingual models perform better than the baselines when averaged across the 10 evaluated language pairs, while handling up to 102 languages

Accepted as a long paper in NAACL 2019


out-of-English. Again, both of the massively multilingual models perform better than the baselines when averaged across the 10 evaluated language pairs, while handling up to 102 languages
to-and-from English and 204 translation tasks simultaneously. In this case the results are similar to those we observed on the TED talks corpus, where the one-to-many model performs better
than the many-to-many model. Again, this advantage may be due to the one-to-many model handling a smaller number of tasks while not being
biased towards English in the target side like the
many-to-many model.
4 Analysis
The above results show that massively multilingual NMT is indeed possible in large scale settings
and can improve performance over strong bilingual baselines. However, it was shown in a somewhat extreme case with more than 100 languages
trained jointly, where we saw that in some cases
the joint training may harm the performance for
some language pairs (i.e. German-English above).
In the following analysis we would like to better understand the trade-off between the number
of languages involved and the translation accuracy while keeping the model capacity and training conﬁguration ﬁxed.
4.1 Multilinguality & Supervised
Performance
We ﬁrst study the effect of varying the number of languages on the translation accuracy in

Accepted as a long paper in NAACL 2019


Ar-Fr Fr-Ar Ru-Uk Uk-Ru Avg.
5-to-5 1.66 4.49 3.7 3.02 3.21
25-to-25 1.83 5.52 16.67 4.31 7.08
50-to-50 4.34 4.72 15.14 20.23 11.1
75-to-75 1.85 4.26 11.2 15.88 8.3
103-to-103 2.87 3.05 12.3 18.49 9.17
Table 8: Zero-Shot performance while varying the
number of languages involved
to-many models. We create four subsets of the
in-house dataset by sub-sampling it to a different number of languages in each subset. In
this way we create four additional English-centric
datasets, containing 5, 25, 50 and 75 languages
each to-and-from English. We make sure that
each subset contains all the languages from the
next smaller subsets – i.e. the 25 language subset contains the 5 language subset, the 50 language subset contains the 25 language subset and
so on. We train a similar-capacity large Transformer model (with 473.7M parameters) on each
of these subsets and measure the performance for
each model on the 8 supervised language pairs

Accepted as a long paper in NAACL 2019


of these subsets and measure the performance for
each model on the 8 supervised language pairs
from the smallest subset – fArabic, French, Russian, Ukrainian g$English. In this way we can
analyze to what extent adding more languages improves or harms translation quality while keeping
the model capacity ﬁxed, testing the capacity vs.
accuracy “saturation point”.
Table 7 shows the results of this experiment,
reporting the test results for the models that performed best on the multilingual development set.
We can see that in most cases the best results
are obtained using the 5-to-5 model, showing that
there is indeed a trade off between the number
of languages and translation accuracy when using a ﬁxed model capacity and the same training setup. One may expect that the gaps between
the different models should become smaller and
even close with more updates, as the models with
more languages see less examples per language
in each batch, thus requiring more updates to improve in terms of BLEU. However, in our setting
these gaps did not close even after the models converged, leaving 2.73 average BLEU difference be-tween the 5-to-5 and the 103-to-103 model.
4.2 Multilinguality & Zero-Shot
Performance
We then study the effect of the number of languages on zero-shot translation accuracy. Since

Accepted as a long paper in NAACL 2019


4.2 Multilinguality & Zero-Shot
Performance
We then study the effect of the number of languages on zero-shot translation accuracy. Since
we ﬁnd zero-shot accuracy as an interesting measure for model generalization, we hypothesize that
by adding more languages, the model is forced to
create a more generalized representation to better utilize its capacity, which may improve zeroshot performance. We choose four language pairs
for this purpose: Arabic $French which are distant languages, and Ukrainian $Russian which are
similar. Table 8 shows the results of our models
on these language pairs. For Arabic $French the
BLEU scores are very low in all cases, with the
50-to-50 and 25-to-25 models being slightly better than rest on Ar-Fr and Fr-Ar respectively. On
Russian $Ukrainian we see clear improvements
when increasing the number of languages to more
than ﬁve.
Figure 2 further illustrates this, showing the better generalization performance of the massively
multilingual models under this zero-shot setting.
While the zero-shot performance in this case is
low and unstable for the 5-to-5 and 25-to-25 modupdate 5-to-5 25-to-25 50-to-50 75-to-75 103-to-103
0 0 0 0 0 0

Accepted as a long paper in NAACL 2019


Accepted as a long paper in NAACL 2019


400000 4.301280901 6.631205231 16.16190225 17.38892049 17.55904406
450000 3.882381693 6.804813445 18.20554733 17.48778224 17.9339543
500000 3.45445089 6.358428299 17.32598543 16.71108902 15.97367823
550000 3.18 6.38 18.65 18.28 17.4
600000 2.86 9.5 18.46 14.92 17.12
650000 2.55 12.2 18.98 15.68 16.19
700000 2.98 8.44 20.16 15.4 18.52
Figure 2: Zero-shot BLEU during training for Ukranian to Russian
els, it is much better for the 50-to-50, 75-to-75 and
103-to-103 models. Given these results we can
say that the balance between capacity and generalization here favors the mid range 50-to-50 model,
even when using models with more than 473M
trained parameters. This may hint at the necessity of even larger models for such settings, which
is a challenging avenue for future work. We also
note that our 103 language corpus includes up to

Accepted as a long paper in NAACL 2019


trained parameters. This may hint at the necessity of even larger models for such settings, which
is a challenging avenue for future work. We also
note that our 103 language corpus includes up to
one million examples per language pair – while in
real-world MT deployments, systems are trained
on much more examples per pair. This again emphasizes the need for better techniques for training
such massively multilingual models as we may already be hitting the capacity barrier in our setting.
5 Related Work
Dong et al. (2015) extended the NMT model of
Bahdanau et al. (2014) to one-to-many translation
(from English into 4 languages) by adding a dedicated decoder per target language, showing improvements over strong single-pair baselines. Firat et al. (2016a,b) proposed many-to-many models (with up to 6 languages) by using separate encoders and decoders per language while sharing
the attention mechanism. They also introduced
the notion of zero-resource translation, where they
use synthetic training data generated through pivoting to train translation directions without available training data. Ha et al. (2016) and Johnson
et al. (2017) proposed to use a shared encoderdecoder-attention model for many-to-many translation (with up to 7 languages in the latter). In
order to determine the target language in such

Accepted as a long paper in NAACL 2019


order to determine the target language in such
scenarios they proposed adding dedicated targetlanguage symbols to the source. This method enabled zero-shot translation, showing the ability of
the model to generalize to unseen pairs.
Recent works propose different methods for parameter sharing between language pairs in multilingual NMT. Blackwood et al. (2018) propose
sharing all parameters but the attention mechanism
and show improvements over sharing all parameters. Sachan and Neubig (2018) explore sharing various components in self-attentional (Transformer) models. Lu et al. (2018) add a shared “interlingua” layer while using separate encoders and
decoders. Zaremoodi et al. (2018) utilize recurrent
units with multiple blocks together with a trainable
routing network. Platanios et al. (2018) propose
to share the entire network, while using a contex-tual parameter generator that learns to generate the
parameters of the system given the desired source
and target languages. Gu et al. (2018) propose
a “Universal Language Representation” layer together with a Mixture-of-Language-Experts component to improve a many-to-one model from 5
languages into English.
While the mentioned studies provide valuable
contributions to improving multilingual models,
they apply their models on only up to 7 languages

Accepted as a long paper in NAACL 2019


languages into English.
While the mentioned studies provide valuable
contributions to improving multilingual models,
they apply their models on only up to 7 languages
(Johnson et al., 2017) and 20 trained directions
(Cettolo et al., 2017) in a single model, whereas
we focus on scaling NMT to much larger numbers of languages and trained directions. Regarding massively multilingual models, Neubig and
Hu (2018) explored methods for rapid adaptation
of NMT to new languages by training multilingual models on the 59-language TED Talks corpus and ﬁne-tuning them using data from the new
languages. While modeling signiﬁcantly more
languages than previous studies, they only train
many-to-one models, which we show are inferior
in comparison to our proposed massively multilingual many-to-many models when evaluated into
English on this dataset.
Tiedemann (2018) trained an English-centric
many-to-many model on translations of the bible
including 927 languages. While this work pointed
to an interesting phenomena in the latent space
learned by the model where it clusters representations of typologically-similar languages together, it did not include any evaluation of the
produced translations. Similarly, Malaviya et al.
(2017) trained a many-to-English system including 1017 languages from bible translations, and

Accepted as a long paper in NAACL 2019


produced translations. Similarly, Malaviya et al.
(2017) trained a many-to-English system including 1017 languages from bible translations, and
used it to infer typological features for the different languages (without evaluating the translation quality). In another relevant work, Artetxe
and Schwenk (2018) trained an NMT model on
93 languages and used the learned representations
to perform cross-lingual transfer learning. Again,
they did not report the performance of the translation model learned in that massively multilingual
setting.
6 Conclusions and Future Work
We showed that NMT models can successfully
scale to 102 languages to-and-from English with
204 trained directions and up to one million examples per direction. Such models improve the
translation quality over similar single-pair baselines when evaluated to and from English by more
than 2 BLEU when averaged over 10 diverse language pairs in each case. We show a similar result on the low-resource TED Talks corpus with 59
languages and 116 trained directions. We analyze
the trade-offs between translation quality and the
number of languages involved, pointing on capacity bottlenecks even with very large models and
showing that massively multilingual models can
generalize better to zero-shot settings.
We hope this work will encourage future research on massively multilingual NMT, enabling

Accepted as a long paper in NAACL 2019


showing that massively multilingual models can
generalize better to zero-shot settings.
We hope this work will encourage future research on massively multilingual NMT, enabling
easier support for systems that can serve more people around the globe. There are many possible avenues for future work, including semi-supervised
learning in such settings, exploring ways to reduce the performance degradation when increasing the number of languages, or using such models
for multilingual transfer learning (McCann et al.,
2017; Eriguchi et al., 2018; Artetxe and Schwenk,
2018). Understanding and improving zero-shot
performance in such scenarios is also a promising
direction for future work.
Acknowledgments
We would like to thank the Google Brain and
Google Translate teams for their useful inputs and
discussions. We would also like to thank the entire
Lingvo development team for their foundational
contributions to this project.
References
Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zeroshot cross-lingual transfer and beyond. arXiv
preprint arXiv:1812.10464 .
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv preprint

Accepted as a long paper in NAACL 2019


Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly
learning to align and translate. arXiv preprint
arXiv:1409.0473 .
Graeme Blackwood, Miguel Ballesteros, and Todd
Ward. 2018. Multilingual neural machine translation with task-speciﬁc attention. In Proceedings of
the 27th International Conference on Computational
Linguistics , Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Ondˇrej Bojar, Rajen Chatterjee, Christian Federmann,
et al. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the
Second Conference on Machine Translation .Ondˇrej Bojar, Rajen Chatterjee, Christian Federmann,
et al. 2016. Findings of the 2016 conference on machine translation. In ACL 2016 FIRST CONFERENCE ON MACHINE TRANSLATION (WMT16) ,
pages 131–198.
Ondˇrej Bojar, Christian Federmann, Mark Fishel,
Yvette Graham, Barry Haddow, Philipp Koehn, and

Accepted as a long paper in NAACL 2019


Ondˇrej Bojar, Christian Federmann, Mark Fishel,
Yvette Graham, Barry Haddow, Philipp Koehn, and
Christof Monz. 2018. Findings of the 2018 conference on machine translation (wmt18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers .
Mauro Cettolo, Federico Marcello, Bentivogli Luisa,
Niehues Jan, St ¨uker Sebastian, Sudoh Katsuitho,
Yoshino Koichiro, and Federmann Christian. 2017.
Overview of the iwslt 2017 evaluation campaign. In
International Workshop on Spoken Language Translation .
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and
Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing .
Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto
Kazawa, and Wolfgang Macherey. 2018. Zeroshot cross-lingual classiﬁcation using multilingual neural machine translation. arXiv preprint
arXiv:1809.04686 .

Accepted as a long paper in NAACL 2019


arXiv:1809.04686 .
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio.
2016a. Multi-way, multilingual neural machine
translation with a shared attention mechanism. In
Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies .
Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan,
Fatos T Yarman Vural, and Kyunghyun Cho. 2016b.
Zero-resource translation with multi-lingual neural
machine translation. In Proceedings of the 2016
Conference on Empirical Methods in Natural Language Processing .
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K.
Li. 2018. Universal neural machine translation for
extremely low resource languages. In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies .
Thanh-Le Ha, Jan Niehues, and Alexander Waibel.
2016. Toward multilingual neural machine translation with universal encoder and decoder. arXiv
preprint arXiv:1611.04798 .
Hany Hassan, Anthony Aue, Chang Chen, et al.

Accepted as a long paper in NAACL 2019


preprint arXiv:1611.04798 .
Hany Hassan, Anthony Aue, Chang Chen, et al.
2018. Achieving human parity on automatic chinese to english news translation. arXiv preprint
arXiv:1803.05567 .
Melvin Johnson, Mike Schuster, Quoc V Le, et al.
2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association of Computational Linguistics.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings
of the 2013 Conference on Empirical Methods in
Natural Language Processing , Seattle, Washington,
USA.
Surafel Melaku Lakew, Mauro Cettolo, and Marcello
Federico. 2018. A comparison of transformer and
recurrent neural networks on multilingual neural
machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA.
Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, and Jason Sun. 2018. A neural interlingua for multilingual machine translation.
InProceedings of the Third Conference on Machine
Translation: Research Papers , Belgium, Brussels.

Accepted as a long paper in NAACL 2019


InProceedings of the Third Conference on Machine
Translation: Research Papers , Belgium, Brussels.
Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017. Learning language representations for typology prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language
Processing .
Bryan McCann, James Bradbury, Caiming Xiong, and
Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems .
Graham Neubig and Junjie Hu. 2018. Rapid adaptation
of neural machine translation to new languages. In
Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing .
Toan Q. Nguyen and David Chiang. 2017. Transfer
learning across low-resource, related languages for
neural machine translation. In Proc. IJCNLP .
Myle Ott, Sergey Edunov, David Grangier, and
Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on
Machine Translation: Research Papers .
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of
40th Annual Meeting of the Association for Computational Linguistics , Philadelphia, Pennsylvania,
USA.

Accepted as a long paper in NAACL 2019


40th Annual Meeting of the Association for Computational Linguistics , Philadelphia, Pennsylvania,
USA.
Emmanouil Antonios Platanios, Mrinmaya Sachan,
Graham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language
Processing .Ye Qi, Sachan Devendra, Felix Matthieu, Padmanabhan Sarguna, and Neubig Graham. 2018. When
and why are pre-trained word embeddings useful for
neural machine translation. In HLT-NAACL .
Devendra Sachan and Graham Neubig. 2018. Parameter sharing methods for multilingual self-attentional
translation models. In Proceedings of the Third
Conference on Machine Translation: Research Papers, Belgium, Brussels.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words with
subword units. In Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume 1, pages
1715–1725.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng
Chen, et al. 2019. Lingvo: a modular and scalable
framework for sequence-to-sequence modeling.

Accepted as a long paper in NAACL 2019


Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng
Chen, et al. 2019. Lingvo: a modular and scalable
framework for sequence-to-sequence modeling.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.
Sequence to sequence learning with neural networks. In Advances in neural information processing systems , pages 3104–3112.
J¨org Tiedemann. 2018. Emerging language spaces
learned from massively multilingual corpora. arXiv
preprint arXiv:1802.00273 .
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in Neural Information Processing Systems , pages 5998–6008.
Xinyi Wang, Hieu Pham, Philip Arthur, and Graham
Neubig. 2019. Multilingual neural machine translation with soft decoupled encoding. In International
Conference on Learning Representations .
Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu,

Accepted as a long paper in NAACL 2019


Conference on Learning Representations .
Yining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu,
and Chengqing Zong. 2018. Three strategies to improve one-to-many multilingual translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , Brussels, Belgium.
Yonghui Wu, Mike Schuster, Zhifeng Chen, et al. 2016.
Google’s neural machine translation system: Bridging the gap between human and machine translation.
arXiv preprint arXiv:1609.08144 .
Poorya Zaremoodi, Wray Buntine, and Gholamreza
Haffari. 2018. Adaptive knowledge sharing in
multi-task learning: Improving low-resource neural
machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational
Linguistics , Melbourne, Australia.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin
Knight. 2016. Transfer learning for low-resource
neural machine translation. In Proceedings of the
2016 Conference on Empirical Methods in Natural
Language Processing .
A Supplementary Material
Language Train set size
Arabic 214111
Hebrew 211819
Russian 208458
Korean 205640
Italian 204503
Japanese 204090
Chinese-Taiwan 202646

Accepted as a long paper in NAACL 2019


Arabic 214111
Hebrew 211819
Russian 208458
Korean 205640
Italian 204503
Japanese 204090
Chinese-Taiwan 202646
Chinese-China 199855
Spanish 196026
French 192304
Portuguese-Brazil 184755
Dutch 183767
Turkish 182470
Romanian 180484
Polish 176169
Bulgarian 174444
Vietnamese 171995
German 167888
Persian 150965
Hungarian 147219
Serbian 136898
Greek 134327
Croatian 122091
Ukrainian 108495
Czech 103093
Thai 98064
Indonesian 87406
Slovak 61470
Swedish 56647
Portuguese 51785
Danish 44940
Albanian 44525
Lithuanian 41919
Macedonian 25335
Finnish 24222
Burmese 21497
Armenian 21360
French-Canadian 19870
Slovenian 19831
Hindi 18798
Norwegian 15825
Georgian 13193
Estonian 10738
Kurdish 10371
Galician 10017
Marathi 9840
Mongolian 7607
Esperanto 6535
Tamil 6224
Urdu 5977
Azerbaijani 5946

Accepted as a long paper in NAACL 2019


Marathi 9840
Mongolian 7607
Esperanto 6535
Tamil 6224
Urdu 5977
Azerbaijani 5946
Bosnian 5664
Chinese 5534
Malay 5220
Basque 5182
Bengali 4649
Belarusian 4509
Kazakh 3317
Table 9: Language pairs in the TED talks dataset (58
languages, paired with English) with the train-set size
for each pair.Languages
Afrikaans Laothian
Albanian Latin
Amharic Latvian
Arabic Lithuanian
Armenian Luxembourgish*
Azerbaijani Macedonian
Basque Malagasy
Belarusian Malay
Bengali Malayalam
Bosnian Maltese
Bulgarian Maori
Burmese Marathi
Catalan Mongolian
Cebuano Nepali
Chichewa* Norwegian
Chinese Pashto
Corsican* Persian
Croatian Polish
Czech Portuguese
Danish Punjabi
Dutch Romanian
Esperanto Russian
Estonian Samoan*
Finnish Scots Gaelic*
French Serbian
Frisian Sesotho
Galician Shona*
Georgian Sindhi*
German Sinhalese
Greek Slovak
Gujarati Slovenian
Haitian Creole Somali

Accepted as a long paper in NAACL 2019


Galician Shona*
Georgian Sindhi*
German Sinhalese
Greek Slovak
Gujarati Slovenian
Haitian Creole Somali
Hausa* Spanish
Hawaiian* Sundanese
Hebrew Swahili
Hindi Swedish
Hmong* Tagalog
Hungarian Tajik*
Icelandic Tamil
Igbo Telugu
Indonesian Thai
Irish Turkish
Italian Ukrainian
Japanese Urdu
Javanese Uzbek
Kannada Vietnamese
Kazakh Welsh
Khmer Xhosa
Korean Yiddish
Kurdish Yoruba*
Kyrgyz Zulu
Table 10: Language pairs in the in-house dataset (102
languages, paired with English). For languages marked
with * we had less than 1M examples, while for the rest
we used exactly 1M.

Accepted as a long paper in NAACL 2019


Attention Augmented Convolutional Networks
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens
Quoc V . Le
Google Brain
fibello,barretzoph,avaswani,shlens,qvl g@google.com
Abstract
Convolutional networks have been the paradigm of
choice in many computer vision applications. The convolution operation however has a signiﬁcant weakness in that it
only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged
as a recent advance to capture long range interactions, but
has mostly been applied to sequence modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to convolutions. We introduce a novel two-dimensional
relative self-attention mechanism that proves competitive
in replacing convolutions as a stand-alone computational
primitive for image classiﬁcation. We ﬁnd in control experiments that the best results are obtained when combining
both convolutions and self-attention. We therefore propose
to augment convolutional operators with this self-attention
mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments show that Attention Augmentation leads

Summary:
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.

Irwan Bello
Barret Zoph
Ashish Vaswani
Jonathon Shlens
Quoc V. Le

ICCV 2019


mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments show that Attention Augmentation leads
to consistent improvements in image classiﬁcation on ImageNet and object detection on COCO across many different models and scales, including ResNets and a stateof-the art mobile constrained network, while keeping the
number of parameters similar. In particular, our method
achieves a 1:3%top-1 accuracy improvement on ImageNet
classiﬁcation over a ResNet50 baseline and outperforms
other attention mechanisms for images such as Squeezeand-Excitation [17]. It also achieves an improvement of
1.4 mAP in COCO Object Detection on top of a RetinaNet
baseline.
1. Introduction
Convolutional Neural Networks have enjoyed tremendous success in many computer vision applications, especially in image classiﬁcation [24, 23]. The design of the
convolutional layer imposes 1) locality via a limited receptive ﬁeld and 2) translation equivariance via weight sharing.
20 30 40 50 60 70
number of parameters (millions)72737475767778798081top-1 accuracy
73.6%76.4%77.9%78.4%

ICCV 2019