number of parameters (millions)72737475767778798081top-1 accuracy
73.6%76.4%77.9%78.4%
74.3%77.5%78.4%78.9%
74.7%77.7%78.7%79.1%
ResNet
SE-ResNet
AA-ResNet (ours)Figure 1. Attention Augmentation systematically improves image classiﬁcation across a large variety of networks of different
scales. ImageNet classiﬁcation accuracy [9] versus the number of
parameters for baseline models (ResNet) [14], models augmented
with channel-wise attention (SE-ResNet) [17] and our proposed
architecture (AA-ResNet).
Both these properties prove to be crucial inductive biases
when designing models that operate over images. However,
the local nature of the convolutional kernel prevents it from
capturing global contexts in an image, often necessary for
better recognition of objects in images [33].
Self-attention [43], on the other hand, has emerged as a
recent advance to capture long range interactions, but has
mostly been applied to sequence modeling and generative
modeling tasks. The key idea behind self-attention is to

ICCV 2019


recent advance to capture long range interactions, but has
mostly been applied to sequence modeling and generative
modeling tasks. The key idea behind self-attention is to
produce a weighted average of values computed from hidden units. Unlike the pooling or the convolutional operator,
the weights used in the weighted average operation are produced dynamically via a similarity function between hidden units. As a result, the interaction between input signals
depends on the signals themselves rather than being predetermined by their relative location like in convolutions. In
particular, this allows self-attention to capture long rangearXiv:1904.09925v5  [cs.CV]  9 Sep 2020
Input Attention maps Weighted average of the values
Standard convolutionOutput
W
H
Nh=2
Values²Head Head¹
Figure 2. Attention-augmented convolution : For each spatial location (h, w), Nhattention maps over the image are computed from
queries and keys. These attention maps are used to compute Nhweighted averages of the values V . The results are then concatenated,
reshaped to match the original volume’s spatial dimensions and mixed with a pointwise convolution. Multi-head attention is applied in
parallel to a standard convolution operation and the outputs are concatenated.
interactions without increasing the number of parameters.

Attention Augmented Convolutional Networks

Summary:
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.

Irwan Bello
Barret Zoph
Ashish Vaswani
Jonathon Shlens
Quoc V. Le

ICCV 2019


parallel to a standard convolution operation and the outputs are concatenated.
interactions without increasing the number of parameters.
In this paper, we consider the use of self-attention for
discriminative visual tasks as an alternative to convolutions. We develop a novel two-dimensional relative selfattention mechanism [37] that maintains translation equivariance while being infused with relative position information, making it well suited for images. Our self-attention
formulation proves competitive for replacing convolutions
entirely, however we ﬁnd in control experiments that the
best results are obtained when combining both. We therefore do not completely abandon the idea of convolutions,
but instead propose to augment convolutions with this selfattention mechanism. This is achieved by concatenating
convolutional feature maps, which enforce locality, to selfattentional feature maps capable of modeling longer range
dependencies (see Figure 2).
We test our method on the CIFAR-100 and ImageNet
classiﬁcation [22, 9] and the COCO object detection [27]
tasks, across a wide range of architectures at different computational budgets, including a state-of-the art resource
constrained architecture [42]. Attention Augmentation
yields systematic improvements with minimal additional

ICCV 2019


constrained architecture [42]. Attention Augmentation
yields systematic improvements with minimal additional
computational burden and notably outperforms the popular Squeeze-and-Excitation [17] channelwise attention approach in all experiments. In particular, Attention Augmentation achieves a 1.3% top-1 accuracy ImageNet on top of
a ResNet50 baseline and 1.4 mAP increase in COCO object detection on top of a RetinaNet baseline. Suprisingly,
experiments also reveal that fully self-attentional models,
a special case of Attention Augmentation, only perform
slightly worse than their fully convolutional counterparts on
ImageNet, indicating that self-attention is a powerful standalone computational primitive for image classiﬁcation.2. Related Work
2.1. Convolutional networks
Modern computer vision has been built on powerful image featurizers learned on image classiﬁcation tasks such
as CIFAR-10 [22] and ImageNet [9]. These datasets have
been used as benchmarks for delineating better image featurizations and network architectures across a broad range
of tasks [21]. For example, improving the “backbone” network typically leads to improvements in object detection
[19] and image segmentation [6]. These observations have
inspired the research and design of new architectures, which
are typically derived from the composition of convolution

ICCV 2019


[19] and image segmentation [6]. These observations have
inspired the research and design of new architectures, which
are typically derived from the composition of convolution
operations across an array of spatial scales and skip connections [23, 41, 39, 40, 14, 47, 13]. Indeed, automated
search strategies for designing architectures based on convolutional primitives result in state-of-the-art accuracy on
large-scale image classiﬁcation tasks that translate across a
range of tasks [55, 21].
2.2. Attention mechanisms in networks
Attention has enjoyed widespread adoption as a computational module for modeling sequences because of its
ability to capture long distance interactions [2, 44, 4, 3].
Most notably, Bahdanau et al. [2] ﬁrst proposed to combine attention with a Recurrent Neural Network [15] for
alignment in Machine Translation. Attention was further
extended by Vaswani et al. [43], where the self-attentional
Transformer architecture achieved state-of-the-art results in
Machine Translation. Using self-attention in cooperation
with convolutions is a theme shared by recent work in Natural Language Processing [49] and Reinforcement Learning [52]. For example, the QANet [50] and Evolved Transformer [38] architectures alternate between self-attention

ICCV 2019


layers and convolution layers for Question Answering applications and Machine Translation respectively. Additionally, multiple attention mechanisms have been proposed for visual tasks to address the weaknesses of convolutions [17, 16, 7, 46, 45, 53]. For instance, Squeezeand-Excitation [17] and Gather-Excite [16] reweigh feature
channels using signals aggregated from entire feature maps,
while BAM [31] and CBAM [46] reﬁne convolutional features independently in the channel and spatial dimensions.
In non-local neural networks [45], improvements are shown
in video classiﬁcation and object detection via the additive use of a few non-local residual blocks that employ
self-attention in convolutional architectures. However, nonlocal blocks are only added to the architecture after ImageNet pretraining and are initialized in such a way that they
do not break pretraining.
In contrast, our attention augmented networks do not rely
on pretraining of their fully convolutional counterparts and
employ self-attention along the entire architecture. The use
of multi-head attention allows the model to attend jointly

ICCV 2019


on pretraining of their fully convolutional counterparts and
employ self-attention along the entire architecture. The use
of multi-head attention allows the model to attend jointly
to both spatial and feature subspaces. Additionally, we enhance the representational power of self-attention over images by extending relative self-attention [37, 18] to two dimensional inputs allowing us to model translation equivariance in a principled way. Finally our method produces additional feature maps, rather than recalibrating convolutional
features via addition [45, 53] or gating [17, 16, 31, 46]. This
property allows us to ﬂexibly adjust the fraction of attentional channels and consider a spectrum of architectures,
ranging from fully convolutional to fully attentional models.
3. Methods
We now formally describe our proposed Attention Augmentation method. We use the following naming conventions:H,WandFinrefer to the height, width and number
of input ﬁlters of an activation map. Nh,dvanddkrespectively refer the number of heads, the depth of values and the
depth of queries and keys in multihead-attention (MHA).
We further assume that Nhdividesdvanddkevenly and
denotedh
vanddh
kthe depth of values and queries/keys per
attention head.

ICCV 2019


We further assume that Nhdividesdvanddkevenly and
denotedh
vanddh
kthe depth of values and queries/keys per
attention head.
3.1. Self-attention over images
Given an input tensor of shape (H;W;F in),1we ﬂatten
it to a matrix X2RHWFinand perform multihead attention as proposed in the Transformer architecture [43]. The
output of the self-attention mechanism for a single head h
1We omit the batch dimension for simplicity.can be formulated as:
Oh=Softmax0
@(XW q)(XW k)T
q
dh
k1
A(XW v)(1)
whereWq,Wk2RFindh
kandWv2RFindh
vare learned
linear transformations that map the input Xto queriesQ=
XW q, keysK=XW kand valuesV=XW v. The outputs
of all heads are then concatenated and projected again as
follows:
MHA (X) =Concath
O1;:::;O Nhi
WO(2)
whereWO2Rdvdvis a learned linear transformation.

ICCV 2019


O1;:::;O Nhi
WO(2)
whereWO2Rdvdvis a learned linear transformation.
MHA (X)is then reshaped into a tensor of shape (H;W;d v)
to match the original spatial dimensions. We note that
multi-head attention incurs a complexity of O((HW)2dk)
and a memory cost of O((HW)2Nh)as it requires to store
attention maps for each head.
3.1.1 Two-dimensional Positional Embeddings
Without explicit information about positions, self-attention
ispermutation equivariant :
MHA ((X)) =(MHA (X))
for any permutation of the pixel locations, making it ineffective for modeling highly structured data such as images. Multiple positional encodings that augment activation
maps with explicit spatial information have been proposed
to alleviate related issues. In particular, the Image Transformer [32] extends the sinusoidal waves ﬁrst introduced in
the original Transformer [43] to 2 dimensional inputs and
CoordConv [29] concatenates positional channels to an activation map.

ICCV 2019


the original Transformer [43] to 2 dimensional inputs and
CoordConv [29] concatenates positional channels to an activation map.
However these encodings did not help in our experiments on image classiﬁcation and object detection (see Section 4.5). We hypothesize that this is because such positional encodings, while not permutation equivariant, do not
satisfy translation equivariance , which is a desirable property when dealing with images. As a solution, we propose
to extend the use of relative position encodings [37] to two
dimensions and present a memory efﬁcient implementation
based on the Music Transformer [18].
Relative positional embeddings: Introduced in [37] for
the purpose of language modeling, relative self-attention
augments self-attention with relative position embeddings
and enables translation equivariance while preventing permutation equivariance. We implement two-dimensional relative self-attention by independently adding relative height
information and relative width information. The attention
logit for how much pixel i= (ix;iy)attends to pixel
j= (jx;jy)is computed as:
li;j=qT
iq
dh
k(kj+rW
jxix+rH
jyiy) (3)

ICCV 2019


li;j=qT
iq
dh
k(kj+rW
jxix+rH
jyiy) (3)
whereqiis the query vector for pixel i(the i-th row of Q),
kjis the key vector for pixel j(the j-th row of K) andrW
jxix
andrH
jyiyare learned embeddings for relative width jxix
and relative height jyiy, respectively. The output of head
hnow becomes:
Oh=Softmax0
@QKT+Srel
H+Srel
Wq
dh
k1
AV (4)
whereSrel
H;Srel
W2RHWHWare matrices of relative position logits along height and width dimensions that satisfy
Srel
H[i;j] =qT
irH
jyiyandSrel
W[i;j] =qT
irW
jxix.
The relative attention algorithm in [37] explicitly
stores all relative embeddings rijin a tensor of shape
(HW;HW;dh
k), thus incurring an additional memory cost
ofO((HW)2dh
k). This compares to O((HW)2Nh)for the

ICCV 2019


k), thus incurring an additional memory cost
ofO((HW)2dh
k). This compares to O((HW)2Nh)for the
position-unaware version self-attention that does not use
position encodings. As we typically have Nh<dh
k, such an
implementation can prove extremely prohibitive and restrict
the number of images that can ﬁt in a minibatch. Instead, we
extend the memory efﬁcient relative masked attention algorithm presented in [18] to unmasked relative self-attention
over 2 dimensional inputs. Our implementation has a memory cost ofO(HWdh
k). We leave the Tensorﬂow code of
the algorithm in the Appendix.
The relative positional embeeddings rHandrWare
learned and shared across heads but not layers. For each
layer, we add (2(H+W)2)dh
kparameters to model relative distances along height and width.
3.2. Attention Augmented Convolution
Multiple previously proposed attention mechanisms over
images [17, 16, 31, 46] suggest that the convolution operator is limited by its locality and lack of understanding
of global contexts. These methods capture long-range dependencies by recalibrating convolutional feature maps. In

ICCV 2019


of global contexts. These methods capture long-range dependencies by recalibrating convolutional feature maps. In
particular, Squeeze-and-Excitation (SE) [17] and GatherExcite (GE) [16] perform channelwise reweighing while
BAM [31] and CBAM [46] reweigh both channels and
spatial positions independently . In contrast to these approaches, we 1) use an attention mechanism that can attend
jointly to spatial and feature subspaces (each head corresponding to a feature subspace) and 2) introduce additional
feature maps rather than reﬁning them. Figure 2 summarizes our proposed augmented convolution.Concatenating convolutional and attentional feature
maps: Formally, consider an original convolution operator with kernel size k,Fininput ﬁlters and Foutoutput
ﬁlters. The corresponding attention augmented convolution
can be written as
AAConv (X) =Concath
Conv (X);MHA (X)i
:
We denote=dv
Foutthe ratio of attentional channels to
number of original output ﬁlters and =dk
Foutthe ratio of
key depth to number of original output ﬁlters. Similarly to

ICCV 2019


number of original output ﬁlters and =dk
Foutthe ratio of
key depth to number of original output ﬁlters. Similarly to
the convolution, the proposed attention augmented convolution 1) is equivariant to translation and 2) can readily operate on inputs of different spatial dimensions. We include
Tensorﬂow code for the proposed attention augmented convolution in the Appendix A.3.
Effect on number of parameters: Multihead attention
introduces a 1x1 convolution with Fininput ﬁlters and
(2dk+dv) =Fout(2+)output ﬁlters to compute queries,
keys and values and an additional 1x1 convolution with
dv=Foutinput and output ﬁlters to mix the contribution of different heads. Considering the decrease in ﬁlters
in the convolutional part, this leads to the following change
in parameters:
paramsFinFout(2+ (1k2)+Fout
Fin2);(5)
where we ignore the parameters introduced by relative position embeddings for simplicity as these are negligible. In
practice, this causes a slight decrease in parameters when
replacing 3x3 convolutions and a slight increase in parameters when replacing 1x1 convolutions. Interestingly, we ﬁnd

ICCV 2019


practice, this causes a slight decrease in parameters when
replacing 3x3 convolutions and a slight increase in parameters when replacing 1x1 convolutions. Interestingly, we ﬁnd
in experiments that attention augmented networks still signiﬁcantly outperform their fully convolutional counterparts
while using less parameters.
Attention Augmented Convolutional Architectures: In
all our experiments, the augmented convolution is followed
by a batch normalization [20] layer which can learn to scale
the contribution of the convolution feature maps and the attention feature maps. We apply our augmented convolution
once per residual block similarly to other visual attention
mechanisms [17, 16, 31, 46] and along the entire architecture as memory permits (see Section 4 for more details).
Since the memory cost O((Nh(HW)2)can be prohibitive for large spatial dimensions, we augment convolutions with attention starting from the last layer (with smallest spatial dimension) until we hit memory constraints. To
reduce the memory footprint of augmented networks, we
typically resort to a smaller batch size and sometimes additionally downsample the inputs to self-attention in the layers with the largest spatial dimensions where it is applied.
Downsampling is performed by applying 3x3 average pooling with stride 2 while the following upsampling (required
for the concatenation) is obtained via bilinear interpolation.

ICCV 2019


Downsampling is performed by applying 3x3 average pooling with stride 2 while the following upsampling (required
for the concatenation) is obtained via bilinear interpolation.
4. Experiments
In the subsequent experiments, we test Attention Augmentation on standard computer vision architectures such
as ResNets [14, 47, 13], and MnasNet [42] on the CIFAR100 [22], ImageNet [9] and COCO [25] datasets. Our experiments show that Attention Augmentation leads to systematic improvements on both image classiﬁcation and object detection tasks across a broad array of architectures and
computational demands. We validate the utility of the proposed two-dimensional relative attention mechanism in ablation experiments. In all experiments, we substitute convolutional feature maps with self-attention feature maps as
it makes for an easier comparison against the baseline models. Unless speciﬁed otherwise, all results correspond to our
two-dimensional relative self-attention mechanism. Experimental details can be found in the Appendix.
4.1. CIFAR-100 image classiﬁcation
We ﬁrst investigate how Attention Augmentation performs on CIFAR-100 [22], a standard benchmark for lowresolution imagery, using a Wide ResNet architecture [51].
The Wide-ResNet-28-10 architecture is comprised of 3

ICCV 2019


The Wide-ResNet-28-10 architecture is comprised of 3
stages of 4 residual blocks each using two 33convolutions. We augment the Wide-ResNet-28-10 by augmenting
the ﬁrst convolution of all residual blocks with relative attention using Nh=8 heads and =2=0.2 and a minimum of
20 dimensions per head for the keys. We compare Attention
Augmentation (AA) against other forms of attention including Squeeze-and-Excitation (SE) [17] and the parameterfree formulation of Gather-Excite (GE) [16]. Table 1 shows
that Attention Augmentation improves performance both
over the baseline network and Squeeze-and-Excitation at a
similar parameter and complexity cost.
Architecture Params GFlops top-1 top-5
Wide-ResNet [51] 36.3M 10.4 80.3 95.0
GE-Wide-ResNet [16] 36.3M 10.4 79.8 95.0
SE-Wide-ResNet [17] 36.5M 10.4 81.0 95.3
AA-Wide-ResNet (ours) 36.2M 10.9 81.6 95.2
Table 1. Image classiﬁcation on the CIFAR-100 dataset [22] using

ICCV 2019


Table 1. Image classiﬁcation on the CIFAR-100 dataset [22] using
the Wide-ResNet 28-10 architecture [51].
4.2. ImageNet image classiﬁcation with ResNet
We next examine how Attention Augmentation performs
on ImageNet [9, 21], a standard large-scale dataset for high
resolution imagery, across an array of architectures. We
start with the ResNet architecture [14, 47, 13] because of itswidespread use and its ability to easily scale across several
computational budgets. The building block in ResNet-34
comprises two 3x3 convolutions with the same number of
output ﬁlters. ResNet-50 and its larger counterparts use a
bottleneck block comprising of 1x1, 3x3, 1x1 convolutions
where the last pointwise convolution expands the number
of ﬁlters and the ﬁrst one contracts the number of ﬁlters.
We modify all ResNets by augmenting the 3x3 convolutions as this decreases number of parameters.2We apply
Attention Augmentation in each residual block of the last 3
stages of the architecture – when the spatial dimensions of

ICCV 2019


Attention Augmentation in each residual block of the last 3
stages of the architecture – when the spatial dimensions of
the activation maps are 28x28, 14x14 and 7x7 – and downsample only during the ﬁrst stage. All attention augmented
networks use =2=0.2, except for ResNet-34 which uses
==0.25. The number of attention heads is ﬁxed to Nh=8.
Architecture Params (M) Infer Train top-1
ResNet-50 25.6 - - 76.4
SE [17] 28.1 +12% +92% 77.5 (77.0)
BAM [31] 25.9 +19% +43% 77.3
CBAM [46] 28.1 +56% +132% 77.4 (77.4)
GALA [28] 29.4 +86% +133% 77.5 (77.3)
AA (= 0:25) 24.3 +29% +25% 77.7
Table 2. Image classiﬁcation performance of different attention
mechanisms on the ImageNet dataset. refers to the increase
in latency times compared to the ResNet50 on a single Tesla V100

ICCV 2019


mechanisms on the ImageNet dataset. refers to the increase
in latency times compared to the ResNet50 on a single Tesla V100
GPU with Tensorﬂow using a batch size of 128. For fair comparison, we also include top-1 results (in parentheses) when scaling
networks in width to match 25:6M parameters as the ResNet50
baseline.
Architecture GFlops Params top-1 top-5
ResNet-34 [14] 7.4 21.8M 73.6 91.5
SE-ResNet-34 [17] 7.4 22.0M 74.3 91.8
AA-ResNet-34 (ours) 7.1 20.7M 74.7 92.0
ResNet-50 [14] 8.2 25.6M 76.4 93.1
SE-ResNet-50 [17] 8.2 28.1M 77.5 93.7
AA-ResNet-50 (ours) 8.3 25.8M 77.7 93.8
ResNet-101 [14] 15.6 44.5M 77.9 94.0
SE-ResNet-101 [17] 15.6 49.3M 78.4 94.2

ICCV 2019


SE-ResNet-101 [17] 15.6 49.3M 78.4 94.2
AA-ResNet-101 (ours) 16.1 45.4M 78.7 94.4
ResNet-152 [14] 23.0 60.2M 78.4 94.2
SE-ResNet-152 [17] 23.1 66.8M 78.9 94.5
AA-ResNet-152 (ours) 23.8 61.6M 79.1 94.6
Table 3. Image classiﬁcation on the ImageNet dataset [9] across
a range of ResNet architectures: ResNet-34, ResNet-50, Resnet101, and ResNet-152 [14, 47, 13].
Table 2 benchmarks Attention Augmentation against
channel and spatial attention mechanisms BAM [31],
CBAM [46] and GALA [28] with channel reduction ratio= 16 on the ResNet50 architecture. Despite the
2We found that augmenting the pointwise expansions works just as well
but does not save parameters or computations.
Architecture GFlops Params top-1 top-5
MnasNet-0.75 0.45 2.91M 73.3 91.3
AA-MnasNet-0.75 0.51 3.02M 73.9 91.6

ICCV 2019


AA-MnasNet-0.75 0.51 3.02M 73.9 91.6
MnasNet-1.0 0.63 3.89M 75.2 92.4
AA-MnasNet-1.0 0.70 4.06M 75.7 92.6
MnasNet-1.25 1.01 5.26M 76.7 93.2
AA-MnasNet-1.25 1.11 5.53M 77.2 93.6
MnasNet-1.4 1.17 6.10M 77.2 93.5
AA-MnasNet-1.4 1.29 6.44M 77.7 93.8
Table 4. Baseline and attention augmented MnasNet [42] accuracies with width multipliers 0.75, 1.0, 1.25 and 1.4.
lack of specialized kernels (See Appendix A.3), Attention
Augmentation offers a competitive accuracy/computational
trade-off compared to previously proposed attention mechanisms. Table 3 compares the non-augmented networks and
Squeeze-and-Excitation (SE) [17] across different network
scales. In all experiments, Attention Augmentation signiﬁcantly increases performance over the non-augmented
baseline and notably outperforms Squeeze-and-Excitation

ICCV 2019


baseline and notably outperforms Squeeze-and-Excitation
(SE) [17] while being more parameter efﬁcient (Figure 1).
Remarkably, our AA-ResNet-50 performs comparably to
the baseline ResNet-101 and our AA-ResNet-101 outperforms the baseline ResNet-152. These results suggest that
attention augmentation is preferable to simply making networks deeper. We include and discuss attention maps visualizations from different pixel positions in the appendix.
4.3. ImageNet classiﬁcation with MnasNet
In this section, we inspect the use of Attention Augmentation in a resource constrained setting by conducting
ImageNet experiments with the MnasNet architecture [42],
which is an extremely parameter-efﬁcient architecture. In
particular, the MnasNet was found by neural architecture search [54], using only the highly optimized mobile inverted bottleneck block [36] and the Squeeze-andExcitation operation [17] as the primitives in its search
space. We apply Attention Augmentation to the mobile
inverted bottleneck by replacing convolutional channels in
the expansion pointwise convolution using =2=0.1 and
Nh=4 heads. Our augmented MnasNets use augmented inverted bottlenecks in the last 13 blocks out of 18 in the

ICCV 2019


Nh=4 heads. Our augmented MnasNets use augmented inverted bottlenecks in the last 13 blocks out of 18 in the
MnasNet architecture, starting when the spatial dimension
is 28x28. We downsample only in the ﬁrst stage where Attention Augmentation is applied. We leave the ﬁnal pointwise convolution, also referred to as the “head”, unchanged.
In Table 4, we report ImageNet accuracies for the baseline MnasNet and its attention augmented variants at different width multipliers. Our experiments show that Attention Augmentation yields accuracy improvements across
all width multipliers. Augmenting MnasNets with relative
self-attention incurs a slight parameter increase, however
2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0
number of parameters (millions)737475767778top-1 accuracy
73.3%75.2%76.7%77.2%
73.9%75.7%77.2%77.7%
MnasNet
AA-MnasNet (ours)Figure 3. ImageNet top-1 accuracy as a function of number of parameters for MnasNet (black) and Attention-Augmented-MnasNet

ICCV 2019


AA-MnasNet (ours)Figure 3. ImageNet top-1 accuracy as a function of number of parameters for MnasNet (black) and Attention-Augmented-MnasNet
(red) with width multipliers 0:75,1:0,1:25and1:4.
we verify in Figure 3 that the accuracy improvements are
not just explained by the parameter increase. Additionally,
we note that the MnasNet architecture employs Squeezeand-Excitation at multiple locations that were optimally selected via architecture search, further suggesting the beneﬁts of our method.
4.4. Object Detection with COCO dataset
We next investigate the use of Attention Augmentation
on the task of object detection on the COCO dataset [27].
We employ the RetinaNet architecture with a ResNet-50
and ResNet-101 backbone as done in [26], using the opensourced RetinaNet codebase.3We apply Attention Augmentation uniquely on the ResNet backbone, modifying
them similarly as in our ImageNet classiﬁcation experiments.
Our relative self-attention mechanism improves the performance of the RetinaNet on both ResNet-50 and ResNet101 as shown in Table 5. Most notably, Attention Augmentation yields a 1.4% mAP improvement over a strong
RetinaNet baseline from [26]. In contrast to the success

ICCV 2019


RetinaNet baseline from [26]. In contrast to the success
of Squeeze-and-Excitation in image classiﬁcation with ImageNet, our experiments show that adding Squeeze-andExcitation operators in the backbone network of the RetinaNet signiﬁcantly hurts performance, in spite of grid
searching over the squeeze ratio 2f4;8;16g. We hypothesize that localization requires precise spatial information which SE discards during the spatial pooling operation,
thereby negatively affecting performance. Self-attention on
the other hand maintains spatial information and is likely to
3https://github.com/tensorflow/tpu/tree/master/
models/official/retinanet
Backbone architecture GFlops Params mAP COCO mAP 50 mAP 75
ResNet-50 [26] 182 33.4M 36.8 54.5 39.5
SE-ResNet-50 [17] 183 35.9M 36.5 54.0 39.1
AA-ResNet-50 (ours) 182 33.1M 38.2 56.5 40.7
ResNet-101 [26] 243 52.4M 38.5 56.4 41.2
SE-ResNet-101 [17] 243 57.2M 37.4 55.0 39.9

ICCV 2019


SE-ResNet-101 [17] 243 57.2M 37.4 55.0 39.9
AA-ResNet-101 (ours) 245 51.7M 39.2 57.8 41.9
Table 5. Object detection on the COCO dataset [27] using the RetinaNet architecture [26] with different backbone architectures. We report
mean Average Precision at three different IoU values.
be able to identify object boundaries successfully. Visualizations of attention maps (See Figures 9 and 10 in the Appendix) reveal that some heads are indeed delineating objects from their background which might be important for
localization.
4.5. Ablation Study
Fully-attentional vision models: In this section, we investigate the performance of Attention Augmentation as a
function of the fraction of attentional channels. As we increase this fraction to 100%, we begin to replace a ConvNet with a fully attentional model, only leaving pointwise
convolutions and the stem unchanged. Table 6 presents the
performance of Attention Augmentation on the ResNet-50
architecture for varying ratios =2f0:25;0:5;0:75;1:0g.
Performance slightly degrades as the ratio of attentional
channels increases, which we hypothesize is partly explained by the average pooling operation for downsampling

ICCV 2019


Performance slightly degrades as the ratio of attentional
channels increases, which we hypothesize is partly explained by the average pooling operation for downsampling
at the ﬁrst stage where Attention Augmentation is applied.
Attention Augmentation proves however quite robust to the
fraction of attentional channels. For instance, AA-ResNet50 with==0.75 outperforms its ResNet-50 counterpart,
while being more parameter and ﬂops efﬁcient, indicating
that mostly employing attentional channels is readily competitive.
Perhaps surprisingly, these experiments also reveal that
our proposed self-attention mechanism is a powerful standalone computational primitive for image classiﬁcation and
that fully attentional models are viable for discriminative visual tasks. In particular, AA-ResNet-50 with ==1, which
uses exclusively attentional channels, is only 2.5% worse
in accuracy than its fully convolutional counterpart, in spite
of downsampling with average pooling and having 25% less
parameters. Notably, this fully attentional architecture4also
outperforms ResNet-34 while being more parameter and
ﬂops efﬁcient (see Table 6).
4We consider pointwise convolutions as dense layers. This architecture

ICCV 2019


ﬂops efﬁcient (see Table 6).
4We consider pointwise convolutions as dense layers. This architecture
employs 4 non-pointwise convolutions in the stem and the ﬁrst stage of the
architecture, but we believe such operations can be replaced by attention
too.Architecture GFlops Params top-1 top-5
ResNet-34 [14] 7.4 21.8M 73.6 91.5
ResNet-50 [14] 8.2 25.6M 76.4 93.1
== 0:25 7.9 24.3M 77.7 93.8
== 0:5 7.3 22.3M 77.3 93.6
== 0:75 6.8 20.7M 76.7 93.2
== 1:0 6.3 19.4M 73.9 91.5
Table 6. Attention Augmented ResNet-50 with varying ratios of
attentional channels.
0 25 50 75 100
fraction of attention versus convolutional channels707274767880top-1 accuracy76.4%77.5%
76.6%
75.4%
71.1%77.7%
77.3%
76.7%
73.9%

ICCV 2019


76.6%
75.4%
71.1%77.7%
77.3%
76.7%
73.9%
no position
with position
Figure 4. Effect of relative position embeddings as the ratio
of attentional channels increases on our Attention-Augmented
ResNet50.
Importance of position encodings: In Figure 4, we show
the effect of our proposed two-dimensional relative position encodings as a function of the fraction of attentional
channels. As expected, experiments demonstrate that our
relative position encodings become increasingly more important as the architecture employs more attentional channels. In particular, the fully self-attentional ResNet-50 gains
2.8% top-1 ImageNet accuracy when using relative position
encodings, which indicates the necessity of maintaining position information for fully self-attentional vision models.
We additionally compare our proposed two-dimensional
Architecture Position Encodings top-1 top-5
AA-ResNet-34 None 74.4 91.9
AA-ResNet-34 2d Sine 74.4 92.0
AA-ResNet-34 CoordConv 74.4 92.0
AA-ResNet-34 Relative (ours) 74.7 92.0
AA-ResNet-50 None 77.5 93.7

ICCV 2019


AA-ResNet-34 Relative (ours) 74.7 92.0
AA-ResNet-50 None 77.5 93.7
AA-ResNet-50 2d Sine 77.5 93.7
AA-ResNet-50 CoordConv 77.5 93.8
AA-ResNet-50 Relative (ours) 77.7 93.8
Table 7. Effects of different position encodings in Attention Augmentation on ImageNet classiﬁcation.
Position Encodings mAP COCO mAP 50 mAP 75
None 37.7 56.0 40.2
CoordConv [29] 37.4 55.5 40.1
Relative (ours) 38.2 56.5 40.7
Table 8. Effects of different position encodings in Attention Augmentation on the COCO object detection task using a RetinaNet
AA-ResNet-50 backbone.
relative position encodings to other position encoding
schemes. We apply Attention Augmentation using the same
hyperparameters as 4.2 with the following different position encoding schemes: 1) The position-unaware version of
self-attention (referred to as None ), 2) a two-dimensional
implementation of the sinusoidal positional waves (referred

ICCV 2019


self-attention (referred to as None ), 2) a two-dimensional
implementation of the sinusoidal positional waves (referred
to as 2d Sine ) as used in [32], 3) CoordConv [29] for which
we concatenate (x,y,r) coordinate channels to the inputs of
the attention function, and 4) our proposed two-dimensional
relative position encodings (referred to as Relative ).
In Table 7 and 8, we present the results on ImageNet
classiﬁcation and the COCO object detection task respectively. On both tasks, Attention Augmentation without position encodings already yields improvements over the fully
convolutional non-augmented variants. Our experiments
also reveal that the sinusoidal encodings and the coordinate
convolution do not provide improvements over the positionunaware version of Attention Augmentation. We obtain additional improvements when using our two-dimensional relative attention, demonstrating the utility of preserving translation equivariance while preventing permutation equivariance.
5. Discussion and future work
In this work, we consider the use of self-attention for vision models as an alternative to convolutions. We introduce
a novel two-dimensional relative self-attention mechanism
for images that enables training of competitive fully selfattentional vision models on image classiﬁcation for the ﬁrst

ICCV 2019


for images that enables training of competitive fully selfattentional vision models on image classiﬁcation for the ﬁrst
time. We propose to augment convolutional operators with
this self-attention mechanism and validate the superiority ofthis approach over other attention schemes. Extensive experiments show that Attention Augmentation leads to systematic improvements on both image classiﬁcation and object detection tasks across a wide range of architectures and
computational settings.
Several open questions from this work remain. In future work, we will focus on the fully attentional regime
and explore how different attention mechanisms trade off
computational efﬁciency versus representational power. For
instance, identifying a local attention mechanism may result in an efﬁcient and scalable computational mechanism
that could prevent the need for downsampling with average
pooling [34]. Additionally, it is plausible that architectural
design choices that are well suited when exclusively relying
on convolutions are suboptimal when using self-attention
mechanisms. As such, it would be interesting to see if using Attention Augmentation as a primitive in automated architecture search procedures proves useful to ﬁnd even better models than those previously found in image classiﬁcation [55], object detection [12], image segmentation [6] and
other domains [5, 1, 35, 8]. Finally, one can ask to which

ICCV 2019


other domains [5, 1, 35, 8]. Finally, one can ask to which
degree fully attentional models can replace convolutional
networks for visual tasks.
Acknowledgements
The authors would like to thank Tsung-Yi Lin, Prajit Ramachandran, Mingxing Tan, Yanping Huang and the
Google Brain team for insightful comments and discussions.
References
[1] Maximilian Alber, Irwan Bello, Barret Zoph, Pieter-Jan Kindermans, Prajit Ramachandran, and Quoc V . Le. Backprop
evolution. CoRR , abs/1808.02822, 2018. 8
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.
Neural machine translation by jointly learning to align and
translate. In International Conference on Learning Representations , 2015. 2
[3] Irwan Bello, Sayali Kulkarni, Sagar Jain, Craig Boutilier,
Ed Huai-hsin Chi, Elad Eban, Xiyang Luo, Alan Mackey,
and Ofer Meshi. Seq2slate: Re-ranking and slate optimization with rnns. CoRR , abs/1810.02019, 2018. 2

ICCV 2019


and Ofer Meshi. Seq2slate: Re-ranking and slate optimization with rnns. CoRR , abs/1810.02019, 2018. 2
[4] Irwan Bello, Hieu Pham, Quoc V . Le, Mohammad Norouzi,
and Samy Bengio. Neural combinatorial optimization with
reinforcement learning. 2016. 2
[5] Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V . Le.
Neural optimizer search with reinforcement learning. In Proceedings of the 34th International Conference on Machine
Learning - Volume 70 , ICML’17, pages 459–468. JMLR.org,
2017. 8
[6] Liang-Chieh Chen, Maxwell Collins, Yukun Zhu, George
Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam,
and Jon Shlens. Searching for efﬁcient multi-scale architectures for dense image prediction. In Advances in Neural
Information Processing Systems , pages 8713–8724, 2018. 2,
8
[7] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng
Yan, and Jiashi Feng. A2-nets: Double attention networks.

ICCV 2019


[7] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng
Yan, and Jiashi Feng. A2-nets: Double attention networks.
CoRR , abs/1810.11579, 2018. 3
[8] Ekin Dogus Cubuk, Barret Zoph, Dandelion Man ´e, Vijay Vasudevan, and Quoc V . Le. Autoaugment: Learning augmentation policies from data. CoRR , abs/1805.09501, 2018. 8
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
and Li Fei-Fei. Imagenet: A large-scale hierarchical image
database. In IEEE Conference on Computer Vision and Pattern Recognition . IEEE, 2009. 1, 2, 5
[10] Xavier Gastaldi. Shake-shake regularization. arXiv preprint
arXiv:1705.07485 , 2017. 11
[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Dropblock:
A regularization method for convolutional networks. In
Advances in Neural Information Processing Systems , pages
10750–10760, 2018. 11

ICCV 2019


A regularization method for convolutional networks. In
Advances in Neural Information Processing Systems , pages
10750–10760, 2018. 11
[12] Golnaz Ghiasi, Tsung-Yi Lin, Ruoming Pang, and Quoc V
Le. NAS-FPN: Learning scalable feature pyramid architecture for object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 8
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.
2, 5
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Identity mappings in deep residual networks. In European
Conference on Computer Vision , 2016. 1, 2, 5, 7, 11
[15] Sepp Hochreiter and Juergen Schmidhuber. Long short-term
memory. Neural Computation , 1997. 2
[16] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea
Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. In Advances in Neural Information
Processing Systems , pages 9423–9433, 2018. 3, 4, 5

ICCV 2019


Processing Systems , pages 9423–9433, 2018. 3, 4, 5
[17] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition , 2018. 1, 2, 3, 4, 5, 6, 7
[18] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M Dai,
Matthew D Hoffman, and Douglas Eck. Music transformer.
InAdvances in Neural Processing Systems , 2018. 3, 4
[19] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu,
Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In
IEEE Conference on Computer Vision and Pattern Recognition, 2017. 2
[20] Sergey Ioffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In International Conference on Learning Representations , 2015. 4

ICCV 2019


Accelerating deep network training by reducing internal covariate shift. In International Conference on Learning Representations , 2015. 4
[21] Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, 2019. 2, 5[22] Alex Krizhevsky. Learning multiple layers of features from
tiny images. Technical report, University of Toronto, 2009.
2, 5
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.
Imagenet classiﬁcation with deep convolutional neural networks. In Advances in Neural Information Processing System, 2012. 1, 2
[24] Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick
Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 1998. 1
[25] Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He,
Bharath Hariharan, and Serge Belongie. Feature pyramid
networks for object detection. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition ,
2017. 5

ICCV 2019


networks for object detection. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition ,
2017. 5
[26] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and
Piotr Doll ´ar. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer
vision , pages 2980–2988, 2017. 6, 7, 11
[27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence
Zitnick. Microsoft coco: Common objects in context. In
European Conference on Computer Vision , pages 740–755.
Springer, 2014. 2, 6, 7
[28] Drew Linsley, Dan Scheibler, Sven Eberhardt, and Thomas
Serre. Global-and-local attention networks for visual recognition. CoRR , abs/1805.08819, 2018. 5
[29] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski
Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An
intriguing failing of convolutional neural networks and the

ICCV 2019


Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An
intriguing failing of convolutional neural networks and the
coordconv solution. In Advances in Neural Information Processing Systems , pages 9628–9639, 2018. 3, 8
[30] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint
arXiv:1608.03983 , 2016. 11
[31] Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In So
Kweon. Bam: bottleneck attention module. In British Machine Vision Conference , 2018. 3, 4, 5
[32] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz
Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine
Learning , 2018. 3, 8
[33] Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos,
Eric Wiewiora, and Serge Belongie. Objects in context.
2007. 1
[34] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan
Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone

ICCV 2019


Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone
self-attention in vision models. CoRR , abs/1906.05909,
2019. 8
[35] Prajit Ramachandran, Barret Zoph, and Quoc V . Le. Searching for activation functions. CoRR , abs/1710.05941, 2017.
8
[36] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted
residuals and linear bottlenecks. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition ,
pages 4510–4520, 2018. 6
[37] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Selfattention with relative position representations. arXiv
preprint arXiv:1803.02155 , 2018. 2, 3, 4
[38] David R. So, Chen Liang, and Quoc V . Le. The evolved
transformer. CoRR , abs/1901.11117, 2019. 3
[39] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and
Alex Alemi. Inception-v4, Inception-Resnet and the impact

ICCV 2019


[39] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and
Alex Alemi. Inception-v4, Inception-Resnet and the impact
of residual connections on learning. In International Conference on Learning Representations Workshop Track , 2016.
2
[40] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,
Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent
Vanhoucke, and Andrew Rabinovich. Going deeper with
convolutions. In IEEE Conference on Computer Vision and
Pattern Recognition , 2015. 2
[41] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon
Shlens, and Zbigniew Wojna. Rethinking the Inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition , 2016. 2, 11
[42] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan,
and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. 2,
5, 6, 11

ICCV 2019


5, 6, 11
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In Advances in Neural
Information Processing Systems , pages 5998–6008, 2017. 1,
2, 3
[44] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer
networks. In NIPS , pages 2692–2700, 2015. 2
[45] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803, 2018. 3
[46] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In
So Kweon. Cbam: Convolutional block attention module.
InProceedings of the European Conference on Computer Vision (ECCV) , pages 3–19, 2018. 3, 4, 5
[47] Saining Xie, Ross Girshick, Piotr Doll ´ar, Zhuowen Tu, and
Kaiming He. Aggregated residual transformations for deep

ICCV 2019


[47] Saining Xie, Ross Girshick, Piotr Doll ´ar, Zhuowen Tu, and
Kaiming He. Aggregated residual transformations for deep
neural networks. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition , 2017. 2, 5
[48] Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and
Koichi Kise. Shakedrop regularization for deep residual
learning. arXiv preprint arXiv:1802.02375 , 2018. 11
[49] Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S.
Chao, and Zhaopeng Tu. Convolutional self-attention network. In CoRR , volume abs/1810.13320, 2018. 2
[50] Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui
Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le.
QAnet: Combining local convolution with global selfattention for reading comprehension. In International Conference on Learning Representations , 2018. 2

ICCV 2019


QAnet: Combining local convolution with global selfattention for reading comprehension. In International Conference on Learning Representations , 2018. 2
[51] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference , 2016. 5[52] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David
Reichert, Timothy Lillicrap, Edward Lockhart, Murray
Shanahan, Victoria Langston, Razvan Pascanu, Matthew
Botvinick, Oriol Vinyals, and Peter Battaglia. Deep reinforcement learning with relational inductive biases. In ICLR ,
2019. 2
[53] Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and
Augustus Odena. Self-attention generative adversarial networks. arXiv:1805.08318 , 2018. 3
[54] Barret Zoph and Quoc V . Le. Neural architecture search
with reinforcement learning. In International Conference on
Learning Representations , 2017. 6
[55] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V
Le. Learning transferable architectures for scalable image

ICCV 2019


[55] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V
Le. Learning transferable architectures for scalable image
recognition. In Proceedings of the IEEE conference on
computer vision and pattern recognition , pages 8697–8710,
2018. 2, 8, 11
A. Appendix
A.1. Experimental details
Tuning Unless speciﬁed otherwise, we use the default hyperparameters found in reference baseline codebases without tuning.was searched inf0.1, 0.2, 0.5g,inf0.0, 0.1,
0.25, 0.5, 0.75, 1.0 gand the number of heads was chosen
based on memory constraints (starting from 8 and decreasing when necessary). We report the ﬁnal accuracy for each
run without performing early stopping.
CIFAR-100 Given the low resolution of CIFAR-100 images, we do not downsample feature maps before the attention operation and instead resort to a smaller batch size.
We train all networks for 500 epochs using synchronous
SGD with momentum 0.9 distributed across 8 TESLA V100
GPUs. The learning rate is linearly scaled from 0 to
0:2B=256, whereBis the total batch size, for the ﬁrst 5%

ICCV 2019


GPUs. The learning rate is linearly scaled from 0 to
0:2B=256, whereBis the total batch size, for the ﬁrst 5%
training epochs and then annealed with cosine decay [30].
We use standard CIFAR preprocessing: mean normalizing,
random ﬂipping and cropping [55, 10, 48]. Non-augmented
architectures are trained with a batch size of 1024 and a
weight decay of 2e-4. Augmented architectures are trained
with batch size of 256 and a weight decay of 5e-4.
ImageNet classiﬁcation with ResNet We train all
ResNet architectures for 100 epochs using synchronous
SGD with momentum 0.9 across 8 TESLA V100 GPUs and
weight decay of 1e-4. We use the largest batch size per
workerB2f32;64;128;256gthat ﬁts in a minibatch. The
initial learning rate is scaled linearly according to the total
batch size using a base learning rate of 0.128 for total batch
size of 256. During training, we linearly scale the learning
rate from 0 to this value for the ﬁrst 5% of training epochs
and divide it by 10 at epochs 30, 60, 80 and 90. We use

ICCV 2019


rate from 0 to this value for the ﬁrst 5% of training epochs
and divide it by 10 at epochs 30, 60, 80 and 90. We use
standard Inception data augmentation as described in [41].
ImageNet classiﬁcation with MnasNet We follow the
training setup described in [42] and train all networks for
350 epochs with the RMSProp optimizer using exponential
learning rate decay. When training our augmented MnasNets, we divide the learning rate by 2 and adjusted the learning rate decay so that the ﬁnal learning rate stays the same.
Object Detection with COCO dataset We follow the
setup described in [26, 11] and train the RetinaNet from
scratch for 150 epochs without using ImageNet pretraining
for the ResNet backbone.
A.2. Computational & Memory costs
Table 9 provides the breakdown of self-attention related computational costs per image. Storing attention
maps in each layer induces a memory cost of Nh(HW)2bfloat16 . At inference, the memory cost for storing attention maps is only 1.2% of the memory required to store
model parameters (49MB).
Layer Memory Params FLOPS
fStage 2 - H=W=14g* 4 600KB 43k 22M

ICCV 2019


model parameters (49MB).
Layer Memory Params FLOPS
fStage 2 - H=W=14g* 4 600KB 43k 22M
fStage 3 - H=W=14g* 6 600KB 90k 40M
fStage 4 - H=W=7g* 3 37.5KB 190k 19M
Training 6MB (total) 1.3M 390M
Inference 600KB (max) 1.3M 390M
Table 9. Computational costs associated with self-attention in the
forward pass of the ResNet50. During inference, we only consider
the largest memory cost since activations are not stored.
Figures 5 and 6 show the accuracies of our attention
augmented networks across FLOPS counts, which correlate
with running times across hardware platforms.
5 10 15 20 25
computational demand (GFlops).72737475767778798081top-1 accuracy.
73.6%76.4%77.9%78.4%
74.3%77.5%78.4%78.9%
74.7%77.7%78.7%79.1%
ResNet
SE-ResNet
AA-ResNet (ours)
Figure 5. ImageNet top-1 accuracy as a function of computational

ICCV 2019


ResNet
SE-ResNet
AA-ResNet (ours)
Figure 5. ImageNet top-1 accuracy as a function of computational
demand for variety of ResNet architectures [14]. From left to right:
ResNet-34, ResNet-50, ResNet-101 and ResNet-152.
0.4 0.6 0.8 1.0 1.2 1.4
computational demand (GFlops).737475767778top-1 accuracy
73.3%75.2%76.7%77.2%
73.9%75.7%77.2%77.7%
MnasNet
AA-MnasNet (ours)
Figure 6. ImageNet top-1 accuracy as a function of computational
demand for MnasNet (black) and Attention-Augmented-MnasNet
(red) with width multipliers 0:75,1:0,1:25and1:4.
A.3. 2D Relative Self-Attention implementation
While our method is simple and only requires matrix
multiplication, addition and the softmax operation (Equations 3 and 4), our implementation relies on non-trivial operations (e.g. tiling, transposing and reshaping) because
no low-level kernels currently exist for hardware platforms.
Future work may develop specialized kernels as previously

ICCV 2019


no low-level kernels currently exist for hardware platforms.
Future work may develop specialized kernels as previously
done for convolutions. Therefore, we believe that current
latency times (Table 2) reﬂect the lack of dedicated engineering as opposed to inefﬁciency in the proposed method.
def shape list(x):
"""Return list of dims, statically where possible."""
static = x.get shape().as list()
shape = tf.shape(x)
ret = []
for i, static dim in enumerate(static):
dim = static dim or shape[i]
ret.append(dim)
return ret
def split heads 2d(inputs, Nh):
"""Split channels into multiple heads."""
B, H, W, d = shape list(inputs)
retshape = [B, H, W, Nh, d // Nh]
split = tf.reshape(inputs, ret shape)
return tf.transpose(split, [0, 3, 1, 2, 4])
def combine heads 2d(inputs):
"""Combine heads (inverse of split heads 2d)."""
transposed = tf.transpose(inputs, [0, 2, 3, 1, 4])
Nh, channels = shape list(transposed)[ 2:]

ICCV 2019


transposed = tf.transpose(inputs, [0, 2, 3, 1, 4])
Nh, channels = shape list(transposed)[ 2:]
retshape = shape list(transposed)[: 2] + [Nh channels]
return tf.reshape(transposed, ret shape)
def rel toabs(x):
"""Converts tensor from relative to aboslute indexing."""
# [B, Nh, L, 2L 1]
B, Nh, L, = shape list(x)
# Pad to shift from relative to absolute indexing.
colpad = tf.zeros((B, Nh, L, 1))
x = tf.concat([x, col pad], axis=3)
flat x = tf.reshape(x, [B, Nh, L 2L])
flat pad = tf.zeros((B, Nh, L 1))
flat xpadded = tf.concat([flat x, flat pad], axis=2)
# Reshape and slice out the padded elements.
final x = tf.reshape(flat xpadded, [B, Nh, L+1, 2 L1])

ICCV 2019


final x = tf.reshape(flat xpadded, [B, Nh, L+1, 2 L1])
final x = final x[:, :, :L, L 1:]
return final x
def relative logits 1d(q, rel k, H, W, Nh, transpose mask):
"""Compute relative logits along one dimenion."""
rellogits = tf.einsum(’bhxyd,md >bhxym’, q, rel k)
# Collapse height and heads
rellogits = tf.reshape(
rellogits, [ 1, Nh H, W, 2 W1])
rellogits = rel toabs(rel logits)
# Shape it and tile height times
rellogits = tf.reshape(rel logits, [ 1, Nh, H, W, W])
rellogits = tf.expand dims(rel logits, axis=3)
rellogits = tf.tile(rel logits, [1, 1, 1, H, 1, 1])
# Reshape for adding to the logits.
rellogits = tf.transpose(rel logits, transpose mask)

ICCV 2019


# Reshape for adding to the logits.
rellogits = tf.transpose(rel logits, transpose mask)
rellogits = tf.reshape(rel logits, [ 1, Nh, H W, HW])
return rel logits
Figure 7. Helper functions in Tensorﬂow for 2D relative selfattention.def relative logits(q, H, W, Nh, dkh):
"""Compute relative logits."""
# Relative logits in width dimension first.
relembeddings w = tf.get variable(
’rwidth’, shape=(2 W1, dkh),
initializer=tf.random normal initializer(dkh 0.5))
# [B, Nh, HW, HW]
rellogits w = relative logits 1d(
q, rel embeddings w, H, W, Nh, [0, 1, 2, 4, 3, 5])
# Relative logits in height dimension next.
# For ease, we 1) transpose height and width,
# 2) repeat the above steps and
# 3) transpose to eventually put the logits
# in their right positions.
relembeddings h = tf.get variable(

ICCV 2019


# 2) repeat the above steps and
# 3) transpose to eventually put the logits
# in their right positions.
relembeddings h = tf.get variable(
’rheight’, shape=(2 H1, dkh),
initializer=tf.random normal initializer(dkh 0.5))
# [B, Nh, HW, HW]
rellogits h = relative logits 1d(
tf.transpose(q, [0, 1, 3, 2, 4]),
relembeddings h, W, H, Nh, [0, 1, 4, 2, 5, 3])
return rel logits h, rel logits w
def self attention 2d(inputs, dk, dv, Nh, relative=True):
"""2d relative self attention."""
, H, W, = shape list(inputs)
dkh = dk // Nh
dvh = dv // Nh
flatten hw = lambda x, d: tf.reshape(x, [ 1, Nh, H W, d])
# Compute q, k, v
kqv = tf.layers.conv2d(inputs, 2 dk + dv, 1)

ICCV 2019


# Compute q, k, v
kqv = tf.layers.conv2d(inputs, 2 dk + dv, 1)
k, q, v = tf.split(kqv, [dk, dk, dv], axis=3)
q= dkh   0.5 # scaled dot product
# After splitting, shape is [B, Nh, H, W, dkh or dvh]
q = split heads 2d(q, Nh)
k = split heads 2d(k, Nh)
v = split heads 2d(v, Nh)
# [B, Nh, HW, HW]
logits = tf.matmul(flatten hw(q, dkh), flatten hw(k, dkh),
transpose b=True)
if relative:
rellogits h, rel logits w = relative logits(q, H, W, Nh,
dkh)
logits += rel logits h
logits += rel logits w
weights = tf.nn.softmax(logits)
attn out = tf.matmul(weights, flatten hw(v, dvh))

ICCV 2019


weights = tf.nn.softmax(logits)
attn out = tf.matmul(weights, flatten hw(v, dvh))
attn out = tf.reshape(attn out, [ 1, Nh, H, W, dvh])
attn out = combine heads 2d(attn out)
# Project heads
attn out = tf.layers.conv2d(attn out, dv, 1)
return attn out
def augmented conv2d(X, Fout, k, dk, dv, Nh, relative):
conv out = tf.layers.conv2d(inputs=X, filters=Fout dv,
kernel size=k, padding=’same’)
attn out = self attention 2d(X, dk, dv, Nh, relative=
relative)
return tf.concat([conv out, attn out], axis=3)
Figure 8. Tensorﬂow code for 2D relative self-attention.
A.4. Attention visualizations.
In Figure 10, we present attention maps visualizations
for the input image shown in Figure 9. We see that attention
heads learn to specialize to different content and notably can
delineate object boundaries.

ICCV 2019


for the input image shown in Figure 9. We see that attention
heads learn to specialize to different content and notably can
delineate object boundaries.
Figure 9. An input image. The red crosses indexed 1 to 4 represent
the pixel locations for which we show the attention maps in Figure
10.
Figure 10. Visualization of attention maps for an augmented convolution in the Attention-Augmented-ResNet50. Rows correspond to the 8 different heads and columns correspond to the 4
pixel locations depicted in the input image (See Figure 9).

ICCV 2019


Learning Temporal Pose Estimation
from Sparsely-Labeled Videos
Gedas Bertasius1;2, Christoph Feichtenhofer1, Du Tran1, Jianbo Shi2, Lorenzo Torresani1
1Facebook AI,2University of Pennsylvania
Abstract
Modern approaches for multi-person pose estimation in video require large amounts
of dense annotations. However, labeling every frame in a video is costly and labor
intensive. To reduce the need for dense annotations, we propose a PoseWarper
network that leverages training videos with sparse annotations (every kframes) to
learn to perform dense temporal pose propagation and estimation. Given a pair
of video frames—a labeled Frame A and an unlabeled Frame B—we train our
model to predict human pose in Frame A using the features from Frame B by
means of deformable convolutions to implicitly learn the pose warping between A
and B. We demonstrate that we can leverage our trained PoseWarper for several
applications. First, at inference time we can reverse the application direction
of our network in order to propagate pose information from manually annotated
frames to unlabeled frames. This makes it possible to generate pose annotations
for the entire video given only a few manually-labeled frames. Compared to
modern label propagation methods based on optical ﬂow, our warping mechanism

Summary:
Modern approaches for multi-person pose estimation in video require large
amounts of dense annotations. However, labeling every frame in a video is
costly and labor intensive. To reduce the need for dense annotations, we
propose a PoseWarper network that leverages training videos with sparse
annotations (every k frames) to learn to perform dense temporal pose
propagation and estimation. Given a pair of video frames---a labeled Frame A
and an unlabeled Frame B---we train our model to predict human pose in Frame A
using the features from Frame B by means of deformable convolutions to
implicitly learn the pose warping between A and B. We demonstrate that we can
leverage our trained PoseWarper for several applications. First, at inference
time we can reverse the application direction of our network in order to
propagate pose information from manually annotated frames to unlabeled frames.
This makes it possible to generate pose annotations for the entire video given
only a few manually-labeled frames. Compared to modern label propagation
methods based on optical flow, our warping mechanism is much more compact (6M
vs 39M parameters), and also more accurate (88.7% mAP vs 83.8% mAP). We also
show that we can improve the accuracy of a pose estimator by training it on an
augmented dataset obtained by adding our propagated poses to the original
manual labels. Lastly, we can use our PoseWarper to aggregate temporal pose
information from neighboring frames during inference. This allows our system to
achieve state-of-the-art pose detection results on the PoseTrack2017 and
PoseTrack2018 datasets. Code has been made available at:
https://github.com/facebookresearch/PoseWarper.

Gedas Bertasius
Christoph Feichtenhofer
Du Tran
Jianbo Shi
Lorenzo Torresani

Accepted to NeurIPS 2019

1. DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection

Gedas Bertasius, Jianbo Shi, Lorenzo Torresani

2. Aggregated Residual Transformations for Deep Neural Networks

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, Kaiming He

3. DeepPose: Human Pose Estimation via Deep Neural Networks

4. Convolutional Random Walk Networks for Semantic Image Segmentation

Gedas Bertasius, Lorenzo Torresani, Stella X. Yu, Jianbo Shi

5. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

6. Mask R-CNN

Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick


for the entire video given only a few manually-labeled frames. Compared to
modern label propagation methods based on optical ﬂow, our warping mechanism
is much more compact ( 6M vs 39M parameters), and also more accurate ( 88:7%
mAP vs 83:8%mAP). We also show that we can improve the accuracy of a
pose estimator by training it on an augmented dataset obtained by adding our
propagated poses to the original manual labels. Lastly, we can use our PoseWarper
to aggregate temporal pose information from neighboring frames during inference.
This allows us to obtain state-of-the-art pose detection results on PoseTrack2017
and PoseTrack2018 datasets. Code has been made available at: https://github.
com/facebookresearch/PoseWarper .
1 Introduction
In recent years, visual understanding methods [ 1–15] have made tremendous progress, partly because
of advances in deep learning [ 16–19], and partly due to the introduction of large-scale annotated
datasets [ 20,21]. In this paper we consider the problem of pose estimation, which has greatly
beneﬁtted from the recent creation of large-scale datasets [ 22,23]. Most of the recent advances
in this area, though, have concentrated on the task of pose estimation in still-images [ 3,23–27].

Accepted to NeurIPS 2019

6. Mask R-CNN


in this area, though, have concentrated on the task of pose estimation in still-images [ 3,23–27].
However, directly applying these image-level models to video is challenging due to nuisance factors
such as motion blur, video defocus, and frequent pose occlusions. Additionally, the process of
collecting annotated pose data in multi-person videos is costly and time consuming. A video typically
contains hundreds of frames that need to be densely-labeled by human annotators. As a result,
datasets for video pose estimation [ 22] are typically smaller and less diverse compared to their image
counterparts [ 21]. This is problematic because modern deep models require large amounts of labeled
data to achieve good performance. At the same time, videos have high informational redundancy as
the content changes little from frame to frame. This raises the question of whether every single frame
in a training video needs to be labeled in order to achieve good pose estimation accuracy.
33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.arXiv:1906.04016v3  [cs.CV]  11 Dec 2019
To reduce the reliance on densely annotated video pose data, in this work, we introduce the PoseWarper network, which operates on sparsely annotated videos, i.e., videos where pose annotations

Accepted to NeurIPS 2019

6. Mask R-CNN


are given only every kframes. Given a pair of frames from the same video—a labeled Frame A
and an unlabeled Frame B—we train our model to detect pose in Frame A, using the features from
Frame B. To achieve this goal, our model leverages deformable convolutions [ 28] across space and
time. Through this mechanism, our model learns to sample features from an unlabeled Frame B to
maximize pose detection accuracy in a labeled Frame A.
Our trained PoseWarper can then be used for several applications. First, we can leverage PoseWarper
to propagate pose information from a few manually-labeled frames across the entire video. Compared
to modern optical ﬂow propagation methods such as FlowNet2 [ 29], our PoseWarper produces more
accurate pose annotations ( 88:7%mAP vs 83:8%mAP), while also employing a much more compact
warping mechanism ( 6M vs 39M parameters). Furthermore, we show that our propagated poses can
serve as effective pseudo labels for training a more accurate pose detector. Finally, our PoseWarper
can be used to aggregate temporal pose information from neighboring frames during inference. This
naturally renders the approach more robust to occlusion or motion blur in individual frames, and leads
to state-of-the-art pose detection results on the PoseTrack2017 and PoseTrack2018 datasets [22].

Accepted to NeurIPS 2019

6. Mask R-CNN


to state-of-the-art pose detection results on the PoseTrack2017 and PoseTrack2018 datasets [22].
2 Related Work
Multi-Person Pose Detection in Images. The traditional approaches for pose estimation leverage
pictorial structures model [ 30–34], which represents human body as a tree-structured graph with
pairwise potentials between the connected body parts. These approaches have been highly successful
in the past, but they tend to fail if some of body parts are occluded. These issues have been partially
addressed by the models that assume a non-tree graph structure [ 35–38]. However, most modern
approaches for single image pose estimation are based on convolutional neural networks [ 3,6,23–
27,39–45]. The method in [ 3] regresses (x;y)joint coordinates directly from the images. More
recent work [ 25] instead predicts pose heatmaps, which leads to an easier optimization problem.
Several approaches [ 24,26,39,46] propose an iterative pose estimation pipeline where the predictions
are reﬁned at different stages inside a CNN or via a recurrent network. The methods in [ 6,23,45]
tackle pose estimation problem in a top-down fashion, ﬁrst detecting bounding boxes of people, and

Accepted to NeurIPS 2019

6. Mask R-CNN


tackle pose estimation problem in a top-down fashion, ﬁrst detecting bounding boxes of people, and
then predicting the pose heatmaps from the cropped images. The work in [ 24] proposes part afﬁnity
ﬁelds module that captures pairwise relationships between different body parts. The approaches
in [42,43] leverage a bottom-up pipeline ﬁrst predicting the keypoints, and then assembling them
into instances. Lastly, a recent work in [ 27], proposes an architecture that preserves high resolution
feature maps, which is shown to be highly beneﬁcial for the multi-person pose estimation task.
Multi-Person Pose Detection in Video. Due to a limited number of large scale benchmarks for
video pose detection, there has been signiﬁcantly fewer methods in the video domain. Several prior
methods [ 22,47,48] tackle a video pose estimation task as a two-stage problem, ﬁrst detecting the
keypoints in individual frames, and then applying temporal smoothing techniques. The method in [ 49]
proposes a spatiotemporal CRF, which is jointly optimized for the pose prediction in video. The
work in [ 50] proposes a personalized video pose estimation framework, which is accomplished by

Accepted to NeurIPS 2019

6. Mask R-CNN


proposes a spatiotemporal CRF, which is jointly optimized for the pose prediction in video. The
work in [ 50] proposes a personalized video pose estimation framework, which is accomplished by
ﬁnetuning the model on a few frames with high conﬁdence keypoints in each video. The approaches
in [51,52] leverage ﬂow based representations for aligning features temporally across multiple
frames, and then using such aligned features for pose detection in individual frames.
In contrast to these prior methods, our primary objective is to learn an effective video pose detector
from sparsely labeled videos. Our approach has similarities to the methods in [ 51,52], which use
ﬂow representations for feature alignment. However, unlike our model, the methods in [ 51,52] do
not optimize their ﬂow representations end-to-end with respect to the pose detection task. As we will
show in our experiments, this is important for strong performance.
3 The PoseWarper Network
Overview. Our goal is to design a model that learns to detect pose from sparsely labeled videos.
Speciﬁcally, we assume that pose annotations in training videos are available every kframes. Inspired
by a recent self-supervised approach for learning facial attribute embeddings [ 53], we formulate the

Accepted to NeurIPS 2019

6. Mask R-CNN


by a recent self-supervised approach for learning facial attribute embeddings [ 53], we formulate the
following task. Given two video frames—a labeled Frame A and an unlabeled Frame B—our model
2
1. Training (Unlabeled      Labeled)
2. Video Pose Propagation (Labeled      Unlabeled)

Accepted to NeurIPS 2019

6. Mask R-CNN


Accepted to NeurIPS 2019

6. Mask R-CNN


Features Relating the          Two Frames
Predicted Pose in Frame A
Forward Pass
Backward Pass

Accepted to NeurIPS 2019

6. Mask R-CNN


Ground Truth Pose Propagated to Frame BFeature Warping
Ground Truth Pose in Frame AFeatures Relating the          Two FramesPose Detection
Feature WarpingFigure 1: A high level overview of our approach for using sparsely labeled videos for pose detection.
Faces in the ﬁgure are artiﬁcially masked for privacy reasons. In each training video, pose annotations
are available only every kframes. During training, our system considers a pair of frames–a labeled
Frame A, and an unlabeled Frame B, and aims to detect pose in Frame A, using the features from
Frame B. Our training procedure is designed to achieve two goals: 1) our model must be able to
extract motion offsets relating these two frames. 2) Using these motion offsets our model must
then be able to rewarp the detected pose heatmap extracted from an unlabeled Frame B in order to
optimize the accuracy of pose detection in a labeled Frame A. After training, we can apply our model
in reverse order to propagate pose information across the entire video from ground truth poses given
for only a few frames.
is allowed to compare Frame A to Frame B but it must predict Pose A (i.e., the pose in Frame A)
using the features from Frame B, as illustrated in Figure 1 (top).

Accepted to NeurIPS 2019

6. Mask R-CNN


using the features from Frame B, as illustrated in Figure 1 (top).
At ﬁrst glance, this task may look overly challenging: how can we predict Pose A by merely using
features from Frame B? However, suppose that we had body joint correspondences between Frame A
and Frame B. In such a scenario, this task would become trivial, as we would simply need to spatially
“warp” the feature maps computed from frame B according to the set of correspondences relating
frame B to frame A. Based on this intuition, we design a learning scheme that achieves two goals: 1)
By comparing Frame A and Frame B, our model must be able to extract motion offsets relating these
two frames. 2) Using these motion offsets our model must be able to rewarp the pose extracted from
an unlabeled Frame B in order to optimize pose detection accuracy in a labeled Frame A.
To achieve these goals, we ﬁrst feed both frames through a backbone CNN that predicts pose heatmaps
for each of the frames. Then, the resulting heatmaps from both frames are used to determine which
points from Frame B should be sampled for detection in Frame A. Finally, the resampled pose
heatmap from Frame B is used to maximize accuracy of Pose A.

Accepted to NeurIPS 2019

6. Mask R-CNN


points from Frame B should be sampled for detection in Frame A. Finally, the resampled pose
heatmap from Frame B is used to maximize accuracy of Pose A.
Backbone Network. Due to its high efﬁciency and accuracy, we use the state-of-the-art High
Resolution Network (HRNet-W48) [ 27] as our backbone CNN. However, we note that our system can
easily integrate other architectures as well. Thus, we envision that future improvements in still-image
pose estimation will further improve the effectiveness of our approach.
Deformable Warping. Initially, we feed Frame A and Frame B through our backbone CNN, which
outputs pose heatmaps fAandfB. Then, we compute the difference  A;B=fAfB. The resulting
feature tensor  A;Bis provided as input to a stack of 33simple residual blocks (as in standard
ResNet-18 or ResNet-34 models), which output a feature tensor A;B. The feature tensor A;Bis
then fed into ﬁve 33convolutional layers each using a different dilation rate d2f3;6;12;18;24g
3
Labeled Frame A (time t)
Offsets

Accepted to NeurIPS 2019

6. Mask R-CNN


Pose      in Frame A A Stack of Residual            3x3 Blocks  Def. Conv. (dilation = 12)   2D Conv. (dilation = 12)   2D Conv. (dilation = 6)

Accepted to NeurIPS 2019

6. Mask R-CNN


2D Conv. (dilation = 24)
  Def. Conv. (dilation = 6)

Accepted to NeurIPS 2019

6. Mask R-CNN


Def. Conv. (dilation = 18)   2D Conv. (dilation = 18)
   2D Conv. (dilation = 3)
Figure 2: An illustration of our PoseWarper architecture. Given a labeled Frame A and an unlabeled
Frame B, which are separated by steps in time, our goal is to detect pose in a labeled Frame A
using the features from an unlabeled Frame B. First, we predict pose heatmaps for both frames. Then,
we compute the difference between pose heatmaps in Frame A and Frame B and feed it through a
stack of 33residual blocks. Afterwards, we attach ﬁve 33convolutional layers with dilation
ratesd2f3;6;12;18;24gand predict ﬁve sets of offsets o(d)(pn)for each pixel location pn. The
predicted offsets are used to rewarp pose heatmap B. All ﬁve rewarped heatmaps are then summed
and the resulting tensor is used to predict pose in Frame A.
to predict ﬁve sets of offsets o(d)(pn)at all pixel locations pn. The motivation for using different
dilation rates at the offset prediction stage comes from the need to consider motion cues at different

Accepted to NeurIPS 2019

6. Mask R-CNN


dilation rates at the offset prediction stage comes from the need to consider motion cues at different
spatial scales. When the body motion is small, a smaller dilation rate may be more useful as it
captures subtle motion cues. Conversely, if the body motion is large, using large dilation rate allows
us to incorporate relevant information further away. Next, the predicted offsets are used to spatially
rewarp the pose heatmap fB. We do this for each of the ﬁve sets of offsets o(d), and then sum up all
ﬁve rewarped pose heatmaps to obtain a ﬁnal output gA;B, which is used to predict pose in Frame A.
We implement the warping mechanism via a deformable convolution [ 28], which takes 1) the offsets
o(d)(pn), and 2) the pose heatmap fBas its inputs, and then outputs a newly sampled pose heatmap
gA;B. The subscript (A;B)is used to indicate that even though gA;Bwas resampled from tensor fB,
the offsets for rewarping were computed using  A;B, which contains information from both frames.
An illustration of our architecture is presented in Figure 2.
Loss Function. As in [ 27], we use a standard pose estimation loss function which computes a mean

Accepted to NeurIPS 2019

6. Mask R-CNN


An illustration of our architecture is presented in Figure 2.
Loss Function. As in [ 27], we use a standard pose estimation loss function which computes a mean
squared error between the predicted, and the ground-truth heatmaps. The ground-truth heatmap is
generated by applying a 2D Gaussian around the location of each joint.
Pose Annotation Propagation. During training, we force our model to warp pose heatmap fBfrom
an unlabeled frame B such that it would match the ground-truth pose heatmap in a labeled Frame A.
Afterwards, we can reverse the application direction of our network. This then, allows us to propagate
pose information from manually annotated frames to unlabeled frames (i.e. from a labeled Frame
A to an unlabeled Frame B). Speciﬁcally, given a pose annotation in Frame A, we can generate its
respective ground-truth heatmap yAby applying a 2D Gaussian around the location of each joint
(identically to how it was done in [ 23,27]. Then, we can predict the offsets for warping ground-truth
heatmapyAto an unlabeled Frame B, from the feature difference  B;A=fBfA. Lastly, we

Accepted to NeurIPS 2019

6. Mask R-CNN


heatmapyAto an unlabeled Frame B, from the feature difference  B;A=fBfA. Lastly, we
use our deformable warping scheme to warp the ground-truth pose heatmap yAto Frame B, thus,
propagating pose annotations to unlabeled frames in the same video. See Figure 1 (bottom) for a
high-level illustration of this scheme.
Temporal Pose Aggregation at Inference Time. Instead of using our model to propagate pose
annotations on training videos, we can also use our deformable warping mechanism to aggregate pose
information from nearby frames during inference in order to improve the accuracy of pose detection.
For every frame at time t, we want to aggregate information from all frames at times t+where
2f 3;2;1;0;1;2;3g. Such a pose aggregation procedure renders pose estimation more robust
to occlusions, motion blur, and video defocus.
4
   Ground Truth Reference             Frame (time t)PoseWarper (time t+1)FlowNet2 (time t+1)
Figure 3: The results of a video pose propagation task by our PoseWarper and FlowNet2 [ 29]. The
ﬁrst frame in each 3-frame sequence illustrates a labeled reference frame at time t. For simplicity,

Accepted to NeurIPS 2019

6. Mask R-CNN


ﬁrst frame in each 3-frame sequence illustrates a labeled reference frame at time t. For simplicity,
we show only the “right ankle” body joint for one person, denoted by a pink circle in each of the
frames (please zoom in for a clearer view). The second frame depicts our propagated “right ankle”
detection from the labeled frame in time t to the unlabeled frame in time t+1. The third frame shows
the propagated detection in frame t+1 produced by the FlowNet2 baseline. In contrast to our method,
FlowNet2 fails to propagate poses when there is large motion, blurriness or occlusions.
Consider a pair of frames, ItandIt+. In this case, we want to use pose information from frame
It+to improve pose detection in frame It. To do this, we ﬁrst feed both frames through our trained
PoseWarper model, and obtain a spatially rewarped (resampled) pose heatmap gt;t+, which is aligned
with respect to frame Itusing the features from frame It+. We can repeat this procedure for every
value, and then aggregate pose information from multiple frames via a summation asP
gt;t+.

Accepted to NeurIPS 2019

6. Mask R-CNN


value, and then aggregate pose information from multiple frames via a summation asP
gt;t+.
Implementation Details. Following the framework in [ 27], for training, we crop a 384288
bounding box around each person and use it as input to our model. During training, we use ground
truth person bounding boxes. We also employ random rotations, scaling, and horizontal ﬂipping to
augment the data. To learn the network, we use the Adam optimizer [ 54] with a base learning rate
of104, which is reduced to 105and106after10, and 15epochs, respectively. The training is
performed using 4Tesla M40 GPUs, and is terminated after 20epochs. We initialize our model with
a HRNet-W48 [ 27] pretrained for a COCO keypoint estimation task. To train the deformable warping
module, we select Frame B, with a random time-gap 2[3;3]relative to Frame A. To compute
features relating the two frames, we use twenty 33residual blocks each with 128channels. Even
though this seems like many convolutional layers, due to a small number of channels in each layer,

Accepted to NeurIPS 2019

6. Mask R-CNN


though this seems like many convolutional layers, due to a small number of channels in each layer,
this amounts to only 5:8M parameters (compared to 39M required to compute optical ﬂow in [ 29]).
To compute the offsets o(d), we use ﬁve 33convolutional layers, each using a different dilation
rate (d= 3;6;12;18;24). To resample the pose heatmap fB, we employ ﬁve 33deformable
convolutional layers, each applied to one of the ﬁve predicted offset maps o(d). The ﬁve deformable
convolution layers too employ different dilation rates of 3;6;12;18;24. During testing, we follow
the same two-stage framework used in [ 27,23]: ﬁrst, we detect the bounding boxes for each person in
the image using the detector in [ 48], and then feed the cropped images to our pose estimation model.
4 Experiments
In this section, we present our results on the PoseTrack [ 22] dataset. We demonstrate the effectiveness
of our approach on three applications: 1) video pose propagation, 2) training a network on annotations
augmented with propagated pose pseudo-labels, 3) temporal pose aggregation during inference.
5

Accepted to NeurIPS 2019

6. Mask R-CNN


augmented with propagated pose pseudo-labels, 3) temporal pose aggregation during inference.
5
Table 1: The results of video pose propagation on the PoseTrack2017 [ 22] validation set (measured in
mAP). We propagate pose information across the entire video from the manual annotations provided
in few frames. To study the effect of different levels of dilated convolutions in our PoseWarper
architecture, we also include several ablation baselines (see the bottom half of the table).
Method Head Shoulder Elbow Wrist Hip Knee Ankle Mean
Pseudo-labeling w/ HRNet [27] 79.1 86.5 81.4 74.7 81.4 79.4 72.3 79.3
Optical Flow Propagation (Farneback [55]) 76.5 82.3 74.3 69.2 80.8 74.8 70.1 75.5
Optical Flow Propagation (FlowNet2 [29]) 82.7 91.0 83.8 78.4 89.7 83.6 78.1 83.8
PoseWarper (no dilated convs) 86.1 91.7 88.0 83.5 90.2 87.3 84.6 87.2

Accepted to NeurIPS 2019

6. Mask R-CNN


PoseWarper (no dilated convs) 86.1 91.7 88.0 83.5 90.2 87.3 84.6 87.2
PoseWarper (1 dilated conv) 85.0 91.6 88.0 83.7 89.6 87.3 84.7 87.0
PoseWarper (2 dilated convs) 85.8 92.4 88.8 84.9 91.0 88.4 86.0 88.0
PoseWarper (3 dilated convs) 86.1 92.6 89.2 85.5 91.3 88.8 86.3 88.4
PoseWarper (4 dilated convs) 86.3 92.6 89.5 85.9 91.9 88.8 86.4 88.6
PoseWarper (5 dilated convs) 86.0 92.7 89.5 86.0 91.5 89.1 86.6 88.7
4.1 Video Pose Propagation
Quantitative Results. To verify that our model learns to capture pose correspondences, we apply it
to the task of video pose propagation, i.e., propagating poses across time from a few labeled frames.
Initially, we train our PoseWarper in a sparsely labeled video setting according to the procedure

Accepted to NeurIPS 2019

6. Mask R-CNN


Initially, we train our PoseWarper in a sparsely labeled video setting according to the procedure
described above. In this setting, every 7thframe of a training video is labeled, i.e. there are 6
unlabeled frames between each pair of manually labeled frames. Since each video contains on
average 30 frames, we have approximately 5annotated frames uniformly spaced out in each video.
Our goal then, is to use our learned PoseWarper to propagate pose annotations from manually-labeled
frames to all unlabeled frames in the same video. Speciﬁcally, for each labeled frame in a video,
we propagate its pose information to the three preceding and three subsequent frames. We train our
PoseWarper on sparsely labeled videos from the training set of PoseTrack2017 [ 22] and then perform
our evaluations on the validation set.
To evaluate the effectiveness of our approach, we compare our model to several relevant baselines.
As our weakest baseline, we use our trained HRNet [ 27] model that simply predicts pose for every
single frame in a video. Furthermore, we also include a few propagation baselines based on warping
annotations using optical ﬂow. The ﬁrst of these uses a standard Farneback optical ﬂow [ 55] to
warp the manually-labeled pose in each labeled frame to its three preceding and three subsequent

Accepted to NeurIPS 2019

6. Mask R-CNN


warp the manually-labeled pose in each labeled frame to its three preceding and three subsequent
frames. We also include a more advanced optical ﬂow propagation baseline that uses FlowNet2
optical ﬂow [29]. Finally, we evaluate our PoseWarper model.
In Table 1, we present our quantitative results for video pose propagation. The evaluation is done
using an mAP metric as in [ 42]. Our best model achieves a 88:7%mAP, while the optical ﬂow
propagation baseline using FlowNet2 [ 29] yields an accuracy of 83:8%mAP. We also note that
compared to the FlowNet2 [ 29] propagation baseline, our PoseWarper warping mechanism is not
only more accurate, but also signiﬁcantly more compact ( 6M vs 39M parameters).
Ablation Studies on Dilated Convolution. In Table 1, we also present the results investigating the
effect of different levels of dilated convolutions in our PoseWarper architecture. We evaluate all these
variants on the task of video pose propagation. First, we report that removing dilated convolution
blocks from the original architecture reduces the accuracy from 88:7mAP to 87:2mAP. We also
note that a network with a single dilated convolution (using a dilation rate of 3) yields 87:0mAP.

Accepted to NeurIPS 2019

6. Mask R-CNN


note that a network with a single dilated convolution (using a dilation rate of 3) yields 87:0mAP.
Adding a second dilated convolution level (using dilation rates of 3;6) improves the accuracy to 88:0.
Three dilation levels (with dilation rates of 3;6;12) yield a mAP of 88:4and four levels (dilation
rates of 3;6;12;18) give a mAP of 88:6. A network with 5dilated convolution levels yields 88:7
mAP. Adding more dilated convolutions does not improve the performance further. Additionally, we
also experimented with two networks that use dilation rates of 1;2;3;4;5, and 4;8;16;24;32, and
report that such models yield mAPs of 88:6and88:5, respectively, which are slightly lower.
Qualitative Comparison to FlowNet2. In Figure 3, we include an illustration of the motion encoded
by PoseWarper, and compare it to the optical ﬂow computed by FlowNet2 for the video pose
propagation task. The ﬁrst frame in each 3-frame sequence illustrates a labeled reference frame at
time t. For a cleaner visualization, we show only the “right ankle” body joint for one person, which is

Accepted to NeurIPS 2019

6. Mask R-CNN


time t. For a cleaner visualization, we show only the “right ankle” body joint for one person, which is
marked with a pink circle in each of the frames. The second frame depicts our propagated “right
6
a) Training HRNet with Propagated Pose Pseudo Labelsb) Temporal Pose Aggregation during Inference0 100 200 300# Labeled Training Videos707580Accuracy (mAP)GT (7x)GT (1x) + pGT (6x)GT (1x)02468# Labeled Frames Per Video707580Accuracy (mAP)GT (7x)GT (1x) + pGT (6x)GT (1x)02468# Labeled Frames Per Video707580Accuracy (mAP)GT (7x)GT (1x) + T-Agg.GT (1x)0 100 200 300# Labeled Training Videos707580Accuracy (mAP)GT (7x)GT (1x) + T-Agg.GT (1x)Figure 4: A ﬁgure illustrating the value of a) training a standard HRNet [ 27] using our propagated
pose pseudo labels (left), and b) our temporal pose aggregation scheme during inference. In both
settings, we study pose detection performance as a function of 1) number of sparsely-labeled training

Accepted to NeurIPS 2019

6. Mask R-CNN


settings, we study pose detection performance as a function of 1) number of sparsely-labeled training
videos (with 1manually-labeled frame per video), and 2) number of labeled frames per video (with
50sparsely-labeled videos in total). All baselines are based on retraining the standard HRNet [ 27]
model on the different training sets. The "GT (1x)" baseline is trained in a standard way on sparsely
labeled video data. The "GT (7x)" baseline uses 7x more manually annotated data relative to the
"GT (1x)" baseline. Our approach on the left subﬁgure ("GT (1x) + pGT (6x)"), augments the
original sparsely labeled video data with our propagated pose pseudo labels ( 6nearby frames for
every manually-labeled frame). Lastly, in b) "GT (1x) + T-Agg" denotes the use of PoseWarper to
fuse pose information from multiple neighboring frames during inference (training is done as in "GT
(1x)" baseline). From the results, we observe that both application modalities of PoseWarper provide
an effective way to achieve strong pose accuracy while reducing the number of manual annotations.
ankle” detection from the labeled frame in time t to the unlabeled frame in time t+1. The third frame

Accepted to NeurIPS 2019

6. Mask R-CNN


ankle” detection from the labeled frame in time t to the unlabeled frame in time t+1. The third frame
shows the propagated detection in frame t+1 produced by the FlowNet2 baseline. These results
suggest that FlowNet2 struggles to accurately warp poses if 1) there is large motion, 2) occlusions, or
3) blurriness. In contrast, our PoseWarper handles these cases robustly, which is also indicated by our
results in Table 1 (i.e., 88:7vs83:8mAP w.r.t. FlowNet2).
4.2 Data Augmentation with PoseWarper
Here we consider the task of propagating poses on sparsely labeled training videos using PoseWarper,
and then using them as pseudo-ground truth labels (in addition to the original manual labels) to
train a standard HRNet-W48 [ 27]. For this experiment, we study the pose detection accuracy as
a function of two variables: 1) the total number of sparsely-labeled videos, and 2) the number of
manually-annotated frames per video. We aim to study how much we can reduce manual labeling
through our mechanism of pose propagation, while maintaining strong pose accuracy. Note, that we
ﬁrst train our PoseWarper on sparsely labeled videos from the training set of PoseTrack2017 [ 22].

Accepted to NeurIPS 2019

6. Mask R-CNN


ﬁrst train our PoseWarper on sparsely labeled videos from the training set of PoseTrack2017 [ 22].
Then, we propagate pose annotations on the same set of training videos. Afterwards, we retrain the
model on the joint training set comprised of sparse manual pose annotations and our propagated
poses. Lastly, we evaluate this trained model on the validation set.
All results are based on a standard HRNet [ 27] model trained on different forms of training data. "GT
(1x)" refers to a model trained on sparsely labeled videos using ground-truth annotations only. "GT
(7x)" baseline employs 7x more manually-annotated poses relative to "GT (1x)" (the annotations
are part of the PoseTrack2017 training set). In comparison, our approach ("GT (1x) + pGT (6x)"),
is trained on a joint training set consisting of sparse manual pose annotations (same as "GT (1x)"
baseline) and our propagated poses (on the training set of PoseTrack2017), which we use as pseudo
ground truth data (pGT). As before, for every labeled frame we propagate the ground truth pose to
the3previous and the 3subsequent frames, which allows us to expand the training set by 7times.

Accepted to NeurIPS 2019

6. Mask R-CNN


the3previous and the 3subsequent frames, which allows us to expand the training set by 7times.
Based on the results in the left subﬁgure of Figure 4, we can draw several conclusions. First, we
note that when there are very few labeled videos (i.e., 5), all three baselines perform poorly (leftmost
ﬁgure). This indicates that in this setting there is not enough data to learn an effective pose detection
model. Second, we observe that when the number of labeled videos is somewhat reasonable (e.g.,
50100), our approach signiﬁcantly outperforms the "GT (1x)" baseline, and is only slightly worse
relative to the "GT (7x)" baseline. As we increase the number of labeled videos, the gaps among the
three methods shrink, suggesting that the model becomes saturated.
7
Table 2: Multi-person pose estimation results on the validation and test sets of PoseTrack2017
and PoseTrack2018 datasets. Even though our model is designed to improve pose detection in
scenarios involving sparsely-labeled videos, here we show that our temporal pose aggregation scheme
during inference is also useful for models trained on densely labeled videos. We improve upon the
state-of-the-art single-frame baselines [23, 27, 56].

Accepted to NeurIPS 2019

6. Mask R-CNN


during inference is also useful for models trained on densely labeled videos. We improve upon the
state-of-the-art single-frame baselines [23, 27, 56].
Dataset Method Head Shoulder Elbow Wrist Hip Knee Ankle Mean
PoseTrack17 Val SetGirdhar et al. [48] 72.8 75.6 65.3 54.3 63.5 60.9 51.8 64.1
Xiu et al. [57] 66.7 73.3 68.3 61.1 67.5 67.0 61.3 66.5
Bin et al [23] 81.7 83.4 80.0 72.4 75.3 74.8 67.1 76.7
HRNet [27] 82.1 83.6 80.4 73.3 75.5 75.3 68.5 77.3
MDPN [56] 85.2 88.5 83.9 77.5 79.0 77.0 71.4 80.7
PoseWarper 81.4 88.3 83.9 78.0 82.4 80.5 73.6 81.2
PoseTrack17 Test SetGirdhar et al. [48] - - - - - - - 59.6

Accepted to NeurIPS 2019

6. Mask R-CNN


Accepted to NeurIPS 2019

6. Mask R-CNN


MDPN [56] - - - 74.5 - - 69.0 76.4
PoseWarper 78.9 84.4 80.9 76.8 75.6 77.5 71.8 78.0
As we vary the number of labeled frames per video (second leftmost ﬁgure), we notice several
interesting patterns. First, we note that for a small number of labeled frames per video (i.e., 12) our
approach outperforms the "GT (1x)" baseline by a large margin. Second, we note that the performance
of our approach and the "GT (7x)" becomes very similar as we add 2or more labeled frames per
video. These ﬁndings further strengthen our previous observation that PoseWarper allows us to reduce
the annotation cost without a signiﬁcant loss in performance.
4.3 Improved Pose Estimation via Temporal Pose Aggregation
In this subsection we assess the ability of PoseWarper to improve the accuracy of pose estimation at
test time by using our deformable warping mechanism to aggregate pose information from nearby
frames. We visualize our results in Figure 4 b), where we evaluate the effectiveness of our temporal
pose aggregation during inference for models trained a) with a different number of labeled videos

Accepted to NeurIPS 2019

6. Mask R-CNN


frames. We visualize our results in Figure 4 b), where we evaluate the effectiveness of our temporal
pose aggregation during inference for models trained a) with a different number of labeled videos
(second rightmost ﬁgure), and b) with a different number of manually-labeled frames per video
(rightmost ﬁgure). We compare our approach ("GT (1x) + T-Agg.") to the same "GT (7x)" and "GT
(1x)" baselines deﬁned in the previous subsection. Note that our method in this case is trained exactly
as "GT (1x)" baseline, the only difference comes from the inference procedure.
When the number of training videos and/or manually labeled frames is small, our approach provides
a signiﬁcant accuracy boost with respect to the "GT (1x)" baseline. However, once, we increase the
number of labeled videos/frames, the gap between all three baselines shrinks, and the model becomes
more saturated. Thus, our temporal pose aggregation scheme during inference is another effective
way to maintain strong performance in a sparsely-labeled video setting.
4.4 Comparison to State-of-the-Art
We also test the effectiveness of our temporal pose aggregation scheme, when the model is trained on

Accepted to NeurIPS 2019

6. Mask R-CNN


4.4 Comparison to State-of-the-Art
We also test the effectiveness of our temporal pose aggregation scheme, when the model is trained on
the full PoseTrack [ 22] dataset. Table 2 compares our method to the most recent approaches in this
area [ 48,57,23,27]. These results suggest that although we designed our method to improve pose
estimation when training videos are sparsely-labeled, our temporal pose aggregation scheme applied
at inference is also useful for models trained on densely-labeled videos. Our PoseWarper obtains
81:2mAP and 77:9mAP on PoseTrack2017 validation and test sets respectively, and 79:7mAP and
78:0mAP on PoseTrack2018 validation and test sets respectively, thus outperforming prior single
frame baselines [48, 57, 23, 27].
8
Frame tFrame t+5Farneback FlowPredicted Human         MotionOffset MagnitudesChannel 99 (x,y)Channel 123 (x,y)
Figure 5: In the ﬁrst two columns, we show a pair of video frames used as input for our model.
The3rdand4thcolumns depict 2randomly selected offset channels visualized as a motion ﬁeld.
Different channels appear to capture the motion of different body parts. In the 5thcolumn, we display

Accepted to NeurIPS 2019

6. Mask R-CNN