framework, where a person tubelet is an instance, and all person tubelets in the
video form a bag. An important consideration in our approach is the presence of
label noise: this is introduced from using o-the-shelf person detectors which have
not been trained on the video-dataset of interest, and also the fact that we have
to sample tubelets from large bags in long videos due to memory constraints. In
both of these scenarios, the standard Multiple Instance Learning assumption [ 7],
that each bag contains at least one instance with the bag-level label, may be
violated. We are not aware of previous work that has explicitly addressed this
problem, and we do so with a probabilistic variant of MIL where we estimate the
uncertainty of an instance-level prediction.
Using our approach, we obtain state-of-the-art results among weakly-supervised
methods on the UCF101-24 dataset. Furthermore, we report, to our knowledge,
the rst weakly-supervised results on the AVA dataset (the only large-scale
dataset for spatio-temporal action recognition), where we also show the accuracy
trade-os when annotating video-clips for time intervals of varying durations.
2 Related Work
Most prior work on spatio-temporal action recognition has been fully-supervised.

ECCV 2020


2 Related Work
Most prior work on spatio-temporal action recognition has been fully-supervised.
Initial approaches in the area used 3D sliding window detectors in conjunction
with handcrafted, volumetric features [ 21,25]. Current state-of-the-art approaches
are temporal extensions of object detection architectures [ 19,41,52,35] such as
Faster-RCNN [ 37] and SSD [ 27]. These approaches predict bounding boxes around
the action in a frame, using as input either a single frame along with optical
Uncertainty-Aware Weakly Supervised Action Detection 3

ow to capture temporal information [ 41,38] or multiple frames at the input to
provide temporal context [ 19]. The predicted bounding boxes are then linked
over time using an online, greedy algorithm or dynamic programming to create
spatio-temporal tracks. Our work builds on these methods by also utilising a
detection architecture and spatio-temporal linking. However, these approaches
all require bounding box annotations at each frame in the video whilst we only
use video-level labels which are signicantly cheaper to acquire.
Weakly supervised approaches to spatio-temporal action recognition have
also been explored before as they enable a signicant reduction in annotation

Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos

Summary:
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.

Anurag Arnab
Chen Sun
Arsha Nagrani
Cordelia Schmid

ECCV 2020


Weakly supervised approaches to spatio-temporal action recognition have
also been explored before as they enable a signicant reduction in annotation
time and cost. Relevant to our approach is the work of [ 6]. Che ron et al. [6] also
use person detections, and infer their action labels using a formulation based on
discriminative clustering [ 2]. Although their approach allows them to incorporate
dierent types of supervision, it eectively learns a linear classier on top of
pretrained, deep features. Our method in contrast is learned fully end-to-end.
Mettes et al. [29] also employed Multiple Instance Learning (MIL), but used
action proposals [ 47] instead of the human detections used by our work and [ 6].
However, [ 29], rely on additional cheap \point" annotations (a single spatial point
annotated for a subset of the frames which constitute the action) which also
ensures that the standard MIL assumption is not violated. In follow-up work [ 28],
the authors removed the need for \point" annotations by incorporating biases
(i.e.the presence of objects in the video, a bias that actions typically occur in the
centre of a frame) instead. Finally, Weinzaepfel et al. [50] also used a Multiple

ECCV 2020


centre of a frame) instead. Finally, Weinzaepfel et al. [50] also used a Multiple
Instance Learning framework in conjunction with human detections. The authors,
however, assumed that sparse spatial supervision was present ( i.e.bounding
boxes for a small subset of frames in the action tube), unlike our method which
requires video-level labels alone.
We also note that many approaches have addressed temporal action detection
(localising actions in time but not space) with only video-level tags as supervision [ 48,42,31,34]. UntrimmedNets [ 48] uses a network with two branches, a
classication module to perform action classication and a selection module to
select relevant frames.Hide-and-Seek [ 42] obtains more precise temporal boundaries by forcing the network to attend to more discriminative frames by randomly
hiding parts of videos. However, these methods are trained and evaluated on
datasets such as ActivityNet [ 4] and THUMOS14 [ 18], which contain mostly one
action per video, and are thus signicantly less challenging than datasets such as
AVA [14] which we evaluate on.
Finally, we note that another approach to combat the eort of dataset
annotation has been various forms of self-supervised learning, where discriminative

ECCV 2020


Finally, we note that another approach to combat the eort of dataset
annotation has been various forms of self-supervised learning, where discriminative
feature representations can be learned with unlabelled data. Examples in video
include cross-modal self-supervised learning by learning correspondences between
the audio and image streams readily available in videos [ 1,33,51], transcribed
speech [ 44] or using meta-data such as hashtags [ 10] as a form of weak labelling.
Self-supervised approaches, however, are complementary to our approach, as they
still require a limited amount of fully-labelled data for the nal task of interest.
4 A. Arnab et al.
In our weakly-supervised action detection scenario, we never have access to full,
spatio-temporal ground-truth annotations for a single training example.
3 Proposed Approach
As shown in Fig. 1, given a set of video clips, with only clip-level annotations of
the actions taking place, our goal is to learn a model to recognise and localise
these actions in space and time. Our method is based on Multiple Instance
Learning (MIL) which we brie
y review in Sec. 3.1. Thereafter, we show how we

ECCV 2020


Learning (MIL) which we brie
y review in Sec. 3.1. Thereafter, we show how we
use it for weakly-supervised spatio-temporal action recognition in Sec. 3.2. We
then describe how the standard MIL assumption, is often violated in our scenario
and describe a method to mitigate this by leveraging uncertainty estimates by
our network in Sec. 3.3. Finally, we discuss implementation details of our network
in Sec. 3.4.
3.1 Multiple Instance Learning
In the standard Multiple Instance Learning (MIL) [ 7] formulation, one is given a
bag ofNinstances, denoted as x=fx1;x2;:::;xNg. The class labels for each
of the instances is unknown, but the label for the entire bag, x, is known. The
standard MIL assumption is that a bag is assigned a class label if at least one
instance in the bag is associated with this label. More formally, we consider the
multi-label classication case, where the label vector for the bag is y2RC, and
yl= 1 if there is at least one instance with the lthlabel is present in the bag,
andyl= 0 otherwise. Note that each bag can be labelled with multiple of the C
class labels.

ECCV 2020


andyl= 0 otherwise. Note that each bag can be labelled with multiple of the C
class labels.
Our goal is to train an instance-level classier (parameterised as a neural
network), that predicts p(yl= 1jxj), or the label probabilities for the jthinstance.
However, as we only have the labels for the entire bag, and not each instance,
MIL methods aggregate the set of instance-level probabilities, fpijgfor a bag
i, to bag-level probabilities, pi, using an aggregation function, g(), where the
probabilities are obtained from a suitable activation function (sigmoid or softmax)
on the logits output by the neural network:
p(yil= 1jx1;x2;:::;xN) =g(pi1;pi2;:::;piN): (1)
Once we have bag-level predictions, we can apply a standard classication
loss between the bag-level probabilities and bag-level ground truth, and train a
neural network with stochastic gradient descent. Since we consider the multi-label
classication case, we use the binary cross-entropy:
Lce(x;y) =NbX
iCX

ECCV 2020


classication case, we use the binary cross-entropy:
Lce(x;y) =NbX
iCX
lyillogpil+ (1yil) log(1pil) (2)
Note that we dened pilas the bag-level probability of the ithbag taking the
lthlabel, which is obtained using Eq. 1, and Nbis the number of bags in the
mini-batch.
Uncertainty-Aware Weakly Supervised Action Detection 5
Aggregation The aggregation function, g(), can naturally be implemented in
neural networks as a global pooling function over all outputs of the network.
Common, permutation-invariant pooling functions include, max-pooling, generalised mean-pooling and log-sum-exponential (LSE) pooling [ 3] (a smooth and
convex approximation of the maximum function) respectively:
g(fpjg) = max
jpj (3)
g(fpjg) =0
@1
jjjX
jpr
j1
A1
r
(4)
g(fpjg) =1
rlog0
@1
jjjX
jerpj1
A (5)

ECCV 2020


A1
r
(4)
g(fpjg) =1
rlog0
@1
jjjX
jerpj1
A (5)
Max-pooling only considers the top-scoring instance in the bag, and thus naturally
captures the MIL assumption that at least one instance in the bag has the
specied, bag-level label. Moreover, it can also be more robust to instances in
the bag that do not have the bag-level label. However, mean and LSE pooling
have been employed in applications such as weakly-supervised segmentation
[36], object recognition [ 45] and medical imaging [ 24] where multiple instances
in the bag do typically have the bag-level label. Note that higher values of
therhyperparameter for both these functions increases their \peakiness" and
approximates the maximum value. For our scenario, detailed in the next section,
we found max-pooling to be the most appropriate.
3.2 Weakly-supervised spatio-temporal action recognition as
multiple instance learning
Our goal is to learn a model to recognise and localise actions in space and
time given only video-level annotations. To facilitate this, we leverage a person
detector that has been trained on a large image dataset, i.e.Microsoft COCO [ 26].

ECCV 2020


time given only video-level annotations. To facilitate this, we leverage a person
detector that has been trained on a large image dataset, i.e.Microsoft COCO [ 26].
Concretely, we run a person detector on our training videos, and create person
tubelets which are person detections over Kconsecutive frames in the video.
Our bag for multiple instance learning thus consists of all the tubelets within a
video, and is annotated with the video-level labels that we have as supervision,
as illustrated in Fig. 2. Note that the size of the bag varies for every video clip, as
the bag size is determined by the length of the video and the number of detected
people.
As shown in Fig. 2, our network architecture for this task is a Fast-RCNN [ 12]
style detector that has been extended temporally. Given a video clip of Kframes,
and proposals which in our case are person detections, the network classies the
action(s) taking place at the centre frame of each proposal, given the temporal
context of the K1 frames around it.
Note that the spatio-temporal localisation task is eectively factorised: the
spatial localisation capability of the model depends on the quality of the person
6 A. Arnab et al.

ECCV 2020


spatial localisation capability of the model depends on the quality of the person
6 A. Arnab et al.
CNN feature extractorROI AlignClassiﬁer
Uncertainty-aware MILLoss ℒMultiple Instance Learning Bag
Probabilities, pUncertainties,  σ
Bag-level label
…
stand, listen to (a person), watch (a person),
walk, carry/hold (an object) …
Fig. 2. Overview of our approach for training an action detector in a weakly-supervised
manner using multiple instance learning: Each bag consists of all the tubelets that have
been extracted from the video-clip. These tubelets are obtained using an o-the-shelf
person detector which has not been trained on the dataset of interest. These tubelets
act as proposals for a Fast-RCNN style detector operating on a sequence of rgb images.
The predictions for each of the tubelets in the bag are then aggregated together, and
compared to the bag-level label. Uncertainty estimates produced by the network are
used to compensate for noise in the bag-level labels during training.
detections. Temporal localisation, on the other hand, is performed by linking
person tubelets through the video as commonly done in the literature [ 19,41,52,6],

ECCV 2020


detections. Temporal localisation, on the other hand, is performed by linking
person tubelets through the video as commonly done in the literature [ 19,41,52,6],
since this method can scale to arbitrarily long videos. We use the same algorithm
as Kalogeiton et al. [19] which links together detections within a small temporal
window greedily based on the spatial intersection over union (IoU) between
bounding boxes on consecutive frames.
Finally, note that for a video consisting of Tframes, the bag could consist of
TK+1 person tubelets if a person is detected on each frame of the video, and a
tubelet is started from each frame. Due to memory limitations, it is infeasible to t
an entire bag onto a GPU for training. As a result, we uniformly sample instances
from each bag during training, whilst still retaining the original bag-level label.
This introduces additional noise into the problem, as detailed next.
3.3 Label noise and violation of the standard MIL assumption
The standard MIL assumption, that at least one instance in the bag is assigned
the bag-level label is often violated in our scenario. There are two primary factors
for this: Firstly, due to computational constraints, we cannot process a whole bag
at a time, but must instead sample instances from a bag. It is therefore possible

ECCV 2020


for this: Firstly, due to computational constraints, we cannot process a whole bag
at a time, but must instead sample instances from a bag. It is therefore possible
to sample a bag that does not contain any tubelets with the labelled action. The
Uncertainty-Aware Weakly Supervised Action Detection 7
Predicted
uncertainty
,

Predicted probability
,

b
a
c
Wrong prediction. Low
uncertainty. Highest loss
Correct prediction. Low
uncertainty. Lowest loss
Incorrect prediction. High uncertainty. Moderate loss
Fig. 3. The loss surface of our uncertainty-based loss (Eq. 6). The ground truth binary
label in this example is 1. Hence, the loss is minimised when the network predicts
a high probability and a low uncertainty (point \a"). However, making an incorrect
prediction with a high uncertainty is not penalised as much (point \b"), and is suitable
for cases when the input bags are noisy and the bag-level label is not present in any of
the tubelets. Finally, predicting the incorrect label with a low uncertainty is penalised
the most (point \c"). Best viewed in colour.
likelihood of this occurring is inversely proportional to the ratio of the duration

ECCV 2020


the most (point \c"). Best viewed in colour.
likelihood of this occurring is inversely proportional to the ratio of the duration
of the labelled action to the total video length. Secondly, in a weakly-supervised
scenario, we use person detectors that are not trained on the video dataset of
interest. Consequently, there can be failures in the detector, especially when
there is a large domain gap between the detector's training distribution and the
video dataset. False negatives (missing detections for people in the scene) are a
particular issue because it is possible that we do not have a single person tubelet
in the bag that corresponds to the labelled action.
Therefore, there are cases when there is no tubelet which actually has the baglevel label. To handle these cases, inspired by [ 22,32], we modify the network to
additionally predict the uncertainty 2RCfor each binary label for all tubelets
in the bag. Intuitively, to minimise the training error, the network can predict the
bag-level label with low uncertainty or it can predict a high uncertainty to avoid
being penalised heavily for noisy bags where the bag-level label is not present in
any of the tubelets. The nal loss, in conjunction with the original cross entropy,
is dened as:
L(x;y; ) =1

ECCV 2020


any of the tubelets. The nal loss, in conjunction with the original cross entropy,
is dened as:
L(x;y; ) =1
2Lce(x;y) + log2(6)
As shown by [ 23], this corresponds to assuming a Boltzmann distribution on the
output of the network with a temperature of 2, and approximately minimising
its log-likelihood.
8 A. Arnab et al.
The loss surface of this probabilistic loss is visualised in Fig. 3. Note how the
loss is the lowest when the predicted label is correct and there is low uncertainty.
However, the loss is not excessive if the incorrect label is predicted with a
high uncertainty. This is in contrast with the standard cross-entropy loss which
penalises incorrect predictions heavily.
3.4 Network architecture and implementation
Our action detector is similar to Fast-RCNN [ 12] using the SlowFast [ 9] video
network architecture based on the ResNet-50 backbone [ 16] pretrained on Kinetics [ 20]. As described in Sec. 3.2, we use region proposals obtained from a
Faster-RCNN detection model trained with Detectron [ 13]. Region-of-interest
features [ 12] are extracted from the last feature map of \res5" using RoIAlign [ 15].

ECCV 2020


features [ 12] are extracted from the last feature map of \res5" using RoIAlign [ 15].
Our choice for this architecture is motivated by the fact that it is simple and has
achieved state-of-the-art results on the AVA dataset [ 14] in a fully-supervised
setting [ 9]. Note that our network does not use additional optical
ow inputs
(which can be considered as an additional source of supervision) as common in
other video architectures [5,19,41,6].
We predict the uncertainty, 2RCfor each of the Cbinary labels dened
by the dataset for each tubelet. As we use max-pooling to aggregate the tubelet
predictions, we select the uncertainty prediction corresponding to the selected
tubelet for computing the loss. For numerical stability, we predict v:=log2
with our network, using the \softplus", f(x) =log(1 + exp(x)), activation
function to ensure positivity. We then compute1
2=exp(v), and avoid the
possibility of dividing by 0 which could be the case if we predicted 2directly
with the network.
We train our network with synchronous stochastic gradient descent (SGD),
using 8 GPUs and a batch size of 4 on each GPU. In our case, each element of a

ECCV 2020


We train our network with synchronous stochastic gradient descent (SGD),
using 8 GPUs and a batch size of 4 on each GPU. In our case, each element of a
batch is of a bag from Multiple Instance Learning. Each bag samples a maximum
of 4 tubelets. Each tubelet itself consists of 16 frames.
4 Experiments
4.1 Experimental set-up
We evaluate our method on UCF101-24 and AVA, described in more detail below.
Note that other video datasets such as THUMOS [ 18] and ActivityNet [ 4] are not
suitable for spatiotemporal localisation, as they lack bounding box annotations.
UCF101-24: UCF101-24 is a subset of the UCF101 [ 43] dataset, consisting of 24
action classes with spatio-temporal localisation annotation, released as bounding
box annotations of humans.Although each video contains only a single action
class, it may contain multiple individuals performing the action with dierent
spatial and temporal boundaries. Moreover, there may also be people present in
the video who are not performing any labelled action. Following standard practice,
Uncertainty-Aware Weakly Supervised Action Detection 9
Table 1. Ablation study of dierent variants of our method on the UCF101-24 validation

ECCV 2020


Uncertainty-Aware Weakly Supervised Action Detection 9
Table 1. Ablation study of dierent variants of our method on the UCF101-24 validation
set. We report the Video mAP at IoU thresholds of 0.2 and 0.5 respectively.
Video AP
0.2 0.5
Weakly supervised baseline 54.3 29.7
MIL - LSE pooling 60.1 33.1
MIL - mean pooling 60.3 33.0
MIL - max pooling 60.7 33.5
MIL - max pooling, uncertainty 61.7 35.0
Fully supervised 69.3 43.6
we use the corrected annotations of [ 41] and report the mean average precision
at a video level (Video AP) for the rst split of the dataset. For evaluating the
Video AP, we link tubelets together using the algorithm of [19].
AVA [ 14]:This dataset consists of 430, 15-minute video clips obtained from
movies. 80 atomic visual actions are annotated exhaustively for all people in
the video, where one person is often simultaneously performing multiple actions.
The dataset annotates keyframes at every second in the video. Following standard practice, we report the Frame AP at an IoU threshold of 0.5 using v2.2
annotations.

ECCV 2020


The dataset annotates keyframes at every second in the video. Following standard practice, we report the Frame AP at an IoU threshold of 0.5 using v2.2
annotations.
4.2 Experiments on UCF101-24
We rst conduct ablation studies of our model on the UCF101-24 dataset. We
discard the spatio-temporal annotations for the whole untrimmed video, and so
our bag in multiple instance learning contains tubelets from the whole video.
Ablation study Table 1 ablates dierent variants of our method: The most na ve
baseline is to not perform any multiple instance learning, and to simply train in
a fully-supervised fashion assuming that the label of a tubelet is the video-level
label. As shown in the rst row of Tab. 1, this method performs the worst as
the assumed tubelet-level labels are often incorrect. The use of multiple instance
learning improves results, with the various aggregation functions performing
similarly. Max-pooling, however, performs the best, and we believe this is because
the max operation is the most suitable for dealing with the noise present in our
tubelets as described in Sec. 3.3. Note that for mean and LSE-pooling, we set
r= 1. Finally, introducing our uncertainty-based loss function improves results

ECCV 2020


tubelets as described in Sec. 3.3. Note that for mean and LSE-pooling, we set
r= 1. Finally, introducing our uncertainty-based loss function improves results
even further, obtaining a Video mAP of 35.0 at a threshold of 0.5. This is 80%
of the performance achieved by our fully-supervised baseline.
Person detections on UCF101-24 Note that for our weakly-supervised experiments, the person tubelets for training are obtained from a Faster-RCNN [ 37]
person detector that has only been trained on Microsoft COCO [ 26]. There is a
10 A. Arnab et al.
signicant domain gap between COCO and UCF, and the annotation protocol of
person boxes on UCF is also not consistent (for example, the bounding box for a
person riding a horse often includes the horse in UCF) with that of COCO. These
discrepancies are re
ected by the fact that our person detections used during
training only have a recall of 46.9% compared to the ground truth person boxes,
when using an IoU threshold of 0.5 to signify a correct match. Furthermore,
the precision of our person tubelets on the training set is only 21.1%. A major
contributing factor to this is that UCF action annotations are not exhaustive

ECCV 2020


the precision of our person tubelets on the training set is only 21.1%. A major
contributing factor to this is that UCF action annotations are not exhaustive
{ there may be people in the video who are not labelled at all as they are not
performing an annotated action. These people will, however, still be detected by
a COCO-trained detector and considered as false positives during this evaluation.
The fact that we are able to train our model with these annotations demonstrates the ability of our multiple instance learning method to handle label noise
in the training set. The inconsistencies in the UCF101-24 dataset labelling are
detailed further in the supplementary, and has also been noted previously by
Ch eron et al. [6].
Noise in the person detections are not a problem for the training of our
fully-supervised baseline, as it is trained with ground-truth boxes in addition
to predicted boxes. As we have box-level supervision in this case, predicted
detections which have an IoU of more than 0.5 with a ground-truth detection
are assigned the label of the ground-truth box, or the negative label otherwise,
during fully-supervised training.
As the goal of this paper is not to develop a better human detector or tracker
for building the person tubelets, we use the Faster-RCNN detector released

ECCV 2020


As the goal of this paper is not to develop a better human detector or tracker
for building the person tubelets, we use the Faster-RCNN detector released
publicly by Ch eron et al. [6] for all our evaluations on the UCF101-24 validation
set. This detector was originally trained on COCO and then netuned on the
UCF101-24 training set using Detectron [13].
The eect of tubelet sampling For the tubelets of length K= 16 that we use,
there is a mean of 33.1 tubelets per video in the UCF101-24 dataset. In computing
this, we only consider tubelets which have a spatio-temporal IoU of less than 0.5
with each other. More tubelets would be obtained if we counted one from each
frame of the video.
As we can t a maximum of 16 tubelets onto a 16GB Nvidia V100 GPU, it
is clear that it is necessary to sample the tubelets in each bag. Note that UCF
videos often have a high number of tubelets, as there are often many people
in the video who are not labelled as performing an action. As described in the
previous subsection, this is also a signicant source of noise.

ECCV 2020


in the video who are not labelled as performing an action. As described in the
previous subsection, this is also a signicant source of noise.
Table 2 shows the eect of changing the batch size (number of bags), and the
number of tubelets sampled per bag, such that GPU memory usage is maximised.
We can see that the uncertainty loss helps in all cases and that accuracy decreases
with low batch sizes. We believe this is due to batch normalisation statistics
being too correlated when more tubes are from the same video.
Comparison to state-of-the-art Table 3 compares our results to the state-of-the-art.
The bottom-half of the table shows that we outperform previous weakly-supervised
Uncertainty-Aware Weakly Supervised Action Detection 11
Table 2. The eect of the number of bags in each training batch on accuracy (Video
AP at 0.5). The uncertainty loss improves accuracy in all scenarios. Although fewer, but
larger, bags can reduce the noise due to sampling, they also cause batch normalisation
statistics to be too correlated, reducing accuracy.
Number of bags in batch Tubelets sampled per bag Video AP without uncertainty Video AP with uncertainty
4 4 33.5 35.0
3 5 33.6 34.1
2 8 33.3 34.2

ECCV 2020


4 4 33.5 35.0
3 5 33.6 34.1
2 8 33.3 34.2
1 16 25.8 26.2
Table 3. Comparison to state-of-the-art methods on the UCF101-24 dataset in both
fully- and weakly-supervised scenarios.
Video AP at 0.2 Video AP at 0.5
Fully supervised
Peng et al. [35] 42.3 35.9
Hou et al. [17] 47.1 {
Weinzaepfel et al. [50] 58.9 {
Saha et al. [38] 63.1 33.1
Singh et al. [41] 73.5 46.3
Zhao et al. [52] 78.5 50.3
Singh et al. [40] 79.0 50.9
Kalogeiton et al. [19] 77.2 51.4
Ours 69.3 43.6
Weakly supervised
Escorcia et al. [8] 45.5 {
Ch eron et al. [6] 43.9 17.7
Ours 61.7 35.0
methods by a large margin. The top-half shows that our fully-supervised baseline

ECCV 2020


Ours 61.7 35.0
methods by a large margin. The top-half shows that our fully-supervised baseline
is also competitive with the fully-supervised state-of-the-art, although that is not
the main goal of this work. The fully-supervised methods which outperform our
method are based on action detectors which directly predict the person proposals
with the network, and are thus able to handle the person annotation peculiarities
of the UCF101-24 dataset more eectively. We do not observe any issues with
person detections for our experiments on AVA in the next section.
Qualitative Results Figure 4 presents qualitative results of our method. The
rst two rows show success cases of our method where the tubelet detection
and linking have performed well. The third row shows a failure case, since the
basketball player represented by the green track is not actually performing the
\Basketball Dunk" action. According to the UCF101-24 annotations, only the
player represented with the blue track is performing this action. This video clip
is thus an example of a video where there are many people not performing the
action annotated for the video, and is especially challenging for our weaklysupervised method. The fourth row shows a dierent failure case as an error by
12 A. Arnab et al.

ECCV 2020


12 A. Arnab et al.
Fig. 4. Qualitative examples on UCF101-24. Note that the bounding boxes are coloured
according to the identity of the track. The action label, and tube score are labelled
from the top-left of the bounding box. Further discussion is included in the text.
the online tubelet linking algorithm (we used the same method as [ 19]) has made
the identities of the two cyclists change after they occluded each other.
4.3 Experiments on AVA
In this section, we report what to our knowledge are the rst weakly-supervised
action detection experiments on AVA [ 14]. The AVA dataset labels keyframes
in a 15-minute video clip, where each keyframe is sampled every second ( i.e.at
1 Hz). The evaluation protocol of the AVA dataset measures the ability of an
action detection model to classify the actions occuring in a keyframe given the
temporal context around it.
We control the diculty of the weakly-supervised action recognition problem by combining the annotations from Nconsecutive keyframes into a single,
clip-level annotation. This eectively means that we are obtaining clip-level

ECCV 2020


clip-level annotation. This eectively means that we are obtaining clip-level
annotations for sub-clips of Nseconds from the original AVA video. The weaklysupervised problem gets more dicult as Nincreases, as the sub-clips get longer
and the number of observed labels within each sub-clip increases. Note that when
N= 1, only the spatial localisation ability of the model is being tested, as during
training, it is unknown which of the subclip-level labels correspond to each person
tubelet in the MIL bag. When N > 1, the subclip-level labels can correspond
Uncertainty-Aware Weakly Supervised Action Detection 13
Table 4. Results of our method on the AVA dataset in terms of the Frame mAP at
an IoU threshold of 0.5. We vary the length of the sub-clips from which we extract
clip-level annotations to control the diculty of the weakly supervised problems. FS
denotes a fully-supervised baseline representing the upper bound on performance. A
sub-clip of 900 seconds is an entire AVA video clip.
Sub-clip duration (seconds)
FS 1 5 10 30 60 900
Frame AP 24.9 22.4 18.0 15.8 11.4 9.1 4.2

ECCV 2020


Sub-clip duration (seconds)
FS 1 5 10 30 60 900
Frame AP 24.9 22.4 18.0 15.8 11.4 9.1 4.2
Table 5. State-of-the-art fully-supervised methods on the AVA dataset.
Method Frame AP
AVA (with optical
ow) [14] 15.6
ARCN (with optical
ow) [46] 17.4
Action Transformer [11] 25.0
SlowFast (ResNet 101) [9] 26.8
SlowFast (ResNet 50, Ours) 24.9
to zero, one or many of the person tubelets at dierent keyframes in the clip,
and it is thus a more dicult task. As an AVA video clip consists of 900 seconds,
N= 900 represents the most extreme case when spatio-temporal annotations are
discarded for the entire 15-minute video.
Table 4 shows the results of our model in this setting. As expected, the
performance of our method improves the shorter the sub-clip. For N= 1 and
N= 5, our method obtains 90% and 72% of fully-supervised performance
respectively, suggesting that bounding-box level annotations are not required for
training action recognition models if the video clips are annotated over short

ECCV 2020


respectively, suggesting that bounding-box level annotations are not required for
training action recognition models if the video clips are annotated over short
temporal intervals. Understandably, the results from N= 900, where we use the
video-level annotations over the whole 15-minute clip are the worst as it is the
most dicult setting.
Figure 5 further analyses the per-class results for the dierent levels of
supervision presented in Tab. 4. As expected, stronger levels of supervision
(shorter sub-clip durations) result in better per-class accuracy. However, some
action classes are aected more than others by weaker labels (longer sub-clips).
Examples of this include \sing to" and \listen to" which show a larger dierence
to the fully-supervised baseline than other classes. Moreover, some classes such
as \watch (a person)", \get up", \close (e.g., a door, a box)" and \hand clap"
perform reasonably when trained with sub-clips ( N10), but much more poorly
when trained with longer sub-clips.
Finally, we compare our fully-supervised baseline to the state-of-the-art in
Tab. 5. Note that our weakly-supervised result from sub-clips of 10 seconds

ECCV 2020


Tab. 5. Note that our weakly-supervised result from sub-clips of 10 seconds
(Tab. 4) outperforms the original fully-supervised baseline using introduced by
the AVA dataset [ 14] that uses both RGB and optical
ow as inputs. Our model,
on the other hand, only uses RGB as its input modality. Our SlowFast model
14 A. Arnab et al.
Fig. 5. Per-class results, in terms of the Frame AP, on the AVA dataset under dierent
levels of supervision (the longer the sub-clip duration, the weaker the supervision). For
clarity, the rst 30 classes, ranked using the fully-supervised model, are shown. As
expected, action classes benet from stronger supervision, while some classes, such as
\watch", \get up" and \close" are very dicult to learn from long sub-clips.
performs similarly to the published results of the original authors [ 9]. Note that
we have not used Non-local [ 49], test-time augmentation or ensembling which
are all complementary methods to improve performance [ 9]. We can see that
in contrast to the UCF dataset in the previous section, our person detector is
accurate on AVA, and so a Fast-RCNN-style detector using person tubelets as

ECCV 2020


in contrast to the UCF dataset in the previous section, our person detector is
accurate on AVA, and so a Fast-RCNN-style detector using person tubelets as
proposals can achieve state-of-the-art results.
5 Conclusion and Future Work
We have proposed a weakly supervised spatio-temporal action detection method
based on Multiple Instance Learning (MIL). Our approach incorporates uncertainty predictions made by the network such that it can better handle noise
in our bags and violations of the standard MIL assumption by predicting a
high uncertainty for noisy bags which cannot be classied correctly. We achieve
state-of-the-art results among weakly supervised methods on the UCF101-24
dataset, and also report the rst weakly-supervised results on AVA, which is the
only large-scale action recognition dataset. Our analysis of the accuracy trade-os
as the time intervals for which sub-clips of the video are annotated will also aid
future dataset annotation eorts.
Future work is to incorporate additional sources of noisy, weakly-labelled
data, such as data which can be scraped o internet search engines.
Uncertainty-Aware Weakly Supervised Action Detection 15
References

ECCV 2020


data, such as data which can be scraped o internet search engines.
Uncertainty-Aware Weakly Supervised Action Detection 15
References
1.Arandjelovic, R., Zisserman, A.: Look, listen and learn. In: Proceedings of the IEEE
International Conference on Computer Vision. pp. 609{617 (2017)
2.Bach, F.R., Harchaoui, Z.: Dirac: a discriminative and
exible framework for
clustering. In: Advances in Neural Information Processing Systems. pp. 49{56
(2008)
3.Boyd, S., Boyd, S.P., Vandenberghe, L.: Convex optimization. Cambridge university
press (2004)
4.Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: A
large-scale video benchmark for human activity understanding. In: Proceedings of
the ieee conference on computer vision and pattern recognition. pp. 961{970 (2015)
5.Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the
kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition. pp. 6299{6308 (2017)

ECCV 2020


kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition. pp. 6299{6308 (2017)
6.Ch eron, G., Alayrac, J.B., Laptev, I., Schmid, C.: A
exible model for training action
localization with varying levels of supervision. In: Advances in Neural Information
Processing Systems. pp. 942{953 (2018)
7.Dietterich, T.G., Lathrop, R.H., Lozano-P erez, T.: Solving the multiple instance
problem with axis-parallel rectangles. Articial intelligence 89(1-2), 31{71 (1997)
8.Escorcia, V., Dao, C.D., Jain, M., Ghanem, B., Snoek, C.: Guess where?
actor-supervision for spatiotemporal action localization. In: arXiv preprint
arXiv:1804.01824 (2018)
9.Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition.
In: Proceedings of the IEEE International Conference on Computer Vision. pp.
6202{6211 (2019)

ECCV 2020


In: Proceedings of the IEEE International Conference on Computer Vision. pp.
6202{6211 (2019)
10.Ghadiyaram, D., Tran, D., Mahajan, D.: Large-scale weakly-supervised pre-training
for video action recognition. In: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition. pp. 12046{12055 (2019)
11.Girdhar, R., Carreira, J., Doersch, C., Zisserman, A.: Video action transformer
network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition. pp. 244{253 (2019)
12. Girshick, R.: Fast r-cnn. In: ICCV (2015)
13.Girshick, R., Radosavovic, I., Gkioxari, G., Doll ar, P., He, K.: Detectron. https:
//github.com/facebookresearch/detectron (2018)
14.Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan,

ECCV 2020


14.Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan,
S., Toderici, G., Ricco, S., Sukthankar, R., et al.: Ava: A video dataset of spatiotemporally localized atomic visual actions. In: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition. pp. 6047{6056 (2018)
15. He, K., Gkioxari, G., Doll ar, P., Girshick, R.: Mask r-cnn. In: ICCV (2017)
16.He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.
In: CVPR (2016)
17.Hou, R., Chen, C., Shah, M.: Tube convolutional neural network (t-cnn) for action
detection in videos. In: International Conference on Computer Vision. pp. 5822{5831
(2017)
18.Jiang, Y.G., Liu, J., Zamir, A.R., Toderici, G., Laptev, I., Shah, M., Sukthankar,
R.: Thumos challenge: Action recognition with a large number of classes (2014)

ECCV 2020


R.: Thumos challenge: Action recognition with a large number of classes (2014)
19.Kalogeiton, V., Weinzaepfel, P., Ferrari, V., Schmid, C.: Joint learning of object
and action detectors. In: Proceedings of the IEEE International Conference on
Computer Vision. pp. 4163{4172 (2017)
16 A. Arnab et al.
20.Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S.,
Viola, F., Green, T., Back, T., Natsev, P., et al.: The kinetics human action video
dataset. arXiv preprint arXiv:1705.06950 (2017)
21.Ke, Y., Sukthankar, R., Hebert, M.: Ecient visual event detection using volumetric
features. In: International Conference on Computer Vision. vol. 1, pp. 166{173.
IEEE (2005)
22.Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning
for computer vision? In: Advances in neural information processing systems. pp.
5574{5584 (2017)

ECCV 2020


for computer vision? In: Advances in neural information processing systems. pp.
5574{5584 (2017)
23.Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh
losses for scene geometry and semantics. In: Computer Vision and Pattern Recognition. pp. 7482{7491 (2018)
24.Kraus, O.Z., Ba, L.J., Frey, B.: Classifying and segmenting microscopy images
using convolutional multiple instance learning. In: arXiv preprint arXiv:1511.05286
(2015)
25.Laptev, I., P erez, P.: Retrieving actions in movies. In: International Conference on
Computer Vision. pp. 1{8. IEEE (2007)
26.Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll ar, P.,
Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV (2014)
27.Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd:

ECCV 2020


Single shot multibox detector. In: European conference on computer vision. pp.
21{37. Springer (2016)
28.Mettes, P., Snoek, C.G., Chang, S.F.: Localizing actions from video labels and
pseudo-annotations. In: British Machine Vision Conference (BMVC) (2017)
29.Mettes, P., Van Gemert, J.C., Snoek, C.G.: Spot on: Action localization from
pointly-supervised proposals. In: European conference on computer vision. pp.
437{453. Springer (2016)
30.Monfort, M., Andonian, A., Zhou, B., Ramakrishnan, K., Bargal, S.A., Yan, T.,
Brown, L., Fan, Q., Gutfruend, D., Vondrick, C., et al.: Moments in time dataset:
one million videos for event understanding. arXiv preprint arXiv:1801.03150 (2018)
31.Nguyen, P., Liu, T., Prasad, G., Han, B.: Weakly supervised action localization
by sparse temporal pooling network. In: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition. pp. 6752{6761 (2018)

ECCV 2020


by sparse temporal pooling network. In: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition. pp. 6752{6761 (2018)
32.Novotny, D., Albanie, S., Larlus, D., Vedaldi, A.: Self-supervised learning of geometrically stable features through probabilistic introspection. In: Computer Vision
and Pattern Recognition. pp. 3637{3645 (2018)
33.Owens, A., Wu, J., McDermott, J.H., Freeman, W.T., Torralba, A.: Ambient sound
provides supervision for visual learning. In: European conference on computer vision.
pp. 801{816. Springer (2016)
34.Paul, S., Roy, S., Roy-Chowdhury, A.K.: W-talc: Weakly-supervised temporal
activity localization and classication. In: Proceedings of the European Conference
on Computer Vision (ECCV). pp. 563{579 (2018)
35.Peng, X., Schmid, C.: Multi-region two-stream r-cnn for action detection. In:
European conference on computer vision. pp. 744{759. Springer (2016)

ECCV 2020


European conference on computer vision. pp. 744{759. Springer (2016)
36.Pinheiro, P.O., Collobert, R.: From image-level to pixel-level labeling with convolutional networks. In: Computer Vision and Pattern Recognition. pp. 1713{1721
(2015)
37.Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing
systems. pp. 91{99 (2015)
Uncertainty-Aware Weakly Supervised Action Detection 17
38.Saha, S., Singh, G., Sapienza, M., Torr, P.H., Cuzzolin, F.: Deep learning for
detecting multiple space-time action tubes in videos. In: BMVC (2016)
39.Sigurdsson, G.A., Russakovsky, O., Gupta, A.: What actions are needed for understanding human actions in videos? In: Proceedings of the IEEE International
Conference on Computer Vision. pp. 2137{2146 (2017)
40.Singh, G., Saha, S., Cuzzolin, F.: Tramnet-transition matrix network for ecient

ECCV 2020


40.Singh, G., Saha, S., Cuzzolin, F.: Tramnet-transition matrix network for ecient
action tube proposals. In: Asian Conference on Computer Vision. pp. 420{437.
Springer (2018)
41.Singh, G., Saha, S., Sapienza, M., Torr, P.H., Cuzzolin, F.: Online real-time multiple
spatiotemporal action localisation and prediction. In: International Conference on
Computer Vision. pp. 3637{3646 (2017)
42.Singh, K.K., Lee, Y.J.: Hide-and-seek: Forcing a network to be meticulous for
weakly-supervised object and action localization. In: 2017 IEEE international
conference on computer vision (ICCV). pp. 3544{3553. IEEE (2017)
43.Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes
from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
44.Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert: A joint model
for video and language representation learning. In: International Conference on

ECCV 2020


for video and language representation learning. In: International Conference on
Computer Vision. pp. 7464{7473 (2019)
45.Sun, C., Paluri, M., Collobert, R., Nevatia, R., Bourdev, L.: Pronet: Learning to
propose object-specic boxes for cascaded neural networks. In: The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR) (June 2016)
46.Sun, C., Shrivastava, A., Vondrick, C., Murphy, K., Sukthankar, R., Schmid, C.:
Actor-centric relation network. In: Proceedings of the European Conference on
Computer Vision (ECCV). pp. 318{334 (2018)
47.Van Gemert, J.C., Jain, M., Gati, E., Snoek, C.G., et al.: Apt: Action localization
proposals from dense trajectories. In: BMVC. vol. 2, p. 4 (2015)
48.Wang, L., Xiong, Y., Lin, D., Van Gool, L.: Untrimmednets for weakly supervised action recognition and detection. In: Proceedings of the IEEE conference on
Computer Vision and Pattern Recognition. pp. 4325{4334 (2017)

ECCV 2020


Computer Vision and Pattern Recognition. pp. 4325{4334 (2017)
49.Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp.
7794{7803 (2018)
50.Weinzaepfel, P., Martin, X., Schmid, C.: Towards weakly-supervised action localization. arXiv preprint arXiv:1605.05197 2(2016)
51.Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J., Torralba, A.:
The sound of pixels. In: Proceedings of the European Conference on Computer
Vision (ECCV). pp. 570{586 (2018)
52.Zhao, J., Snoek, C.G.: Dance with
ow: Two-in-one stream action detection. In:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
pp. 9935{9944 (2019)

ECCV 2020


Is Space-Time Attention All You Need for Video Understanding?
Gedas Bertasius1Heng Wang1Lorenzo Torresani1 2
Abstract
We present a convolution-free approach to video
classiﬁcation built exclusively on self-attention
over space and time. Our method, named “TimeSformer,” adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of framelevel patches. Our experimental study compares different self-attention schemes and suggests that “divided attention,” where temporal
attention and spatial attention are separately applied within each block, leads to the best video
classiﬁcation accuracy among the design choices
considered. Despite the radically new design,
TimeSformer achieves state-of-the-art results on
several action recognition benchmarks, including the best reported accuracy on Kinetics-400
and Kinetics-600. Finally, compared to 3D
convolutional networks, our model is faster to
train, it can achieve dramatically higher test
efﬁciency (at a small drop in accuracy), and
it can also be applied to much longer video
clips (over one minute long). Code and models are available at: https://github.com/
facebookresearch/TimeSformer .
1. Introduction

Summary:
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.

Gedas Bertasius
Heng Wang
Lorenzo Torresani

Accepted to ICML 2021

Quantifying Attention Flow in Transformers

Layer Normalization

Lei Jimmy Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Attention Augmented Convolutional Networks

Bello, I., Zoph, B., Le, Q., Vaswani, A., Shlens, J.

Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation

Language Models are Few-Shot Learners


clips (over one minute long). Code and models are available at: https://github.com/
facebookresearch/TimeSformer .
1. Introduction
Over the last few years, the ﬁeld of natural language processing (NLP) has been revolutionized by the emergence of
methods based on self-attention (Vaswani et al., 2017a). Because of their excellent capabilities at capturing long-range
dependencies among words as well as their training scalability, self-attention architectures, such as the Transformer
model, represent the current state-of-the-art across a wide
range of language tasks, including machine translation (Ott
et al., 2018; Chen et al., 2018a), question answering (Devlin et al., 2019; Dai et al., 2019), and autoregressive word
generation (Radford et al., 2019; Brown et al., 2020).
Video understanding shares several high-level similarities
1Facebook AI2Dartmouth College. Correspondence to: Gedas
Bertasius <gberta@seas.upenn.edu>.with NLP. First of all, videos and sentences are both sequential. Furthermore, precisely as the meaning of a word can
often be understood only by relating it to the other words in
the sentence, it may be argued that atomic actions in shortterm segments need to be contextualized with the rest of the

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


often be understood only by relating it to the other words in
the sentence, it may be argued that atomic actions in shortterm segments need to be contextualized with the rest of the
video in order to be fully disambiguated. Thus, one would
expect the long-range self-attention models from NLP to be
highly effective for video modeling as well. However, in the
video domain, 2D or 3D convolutions still represent the core
operators for spatiotemporal feature learning across different video tasks (Feichtenhofer et al., 2019a; Teed & Deng,
2020; Bertasius & Torresani, 2020). While self-attention
has shown beneﬁts when applied on top of convolutional
layers (Wang et al., 2018a), to the best of our knowledge, no
attempt to use self-attention as the exclusive building block
for video recognition models has been reported.
In this work we pose the question of whether it may be
possible to build a performant convolution-free video architecture by replacing altogether the convolution operator with
self-attention. We argue that such a design has the potential to overcome a few inherent limitations of convolutional
models for video analysis. First, while their strong inductive
biases (e.g., local connectivity and translation equivariance)

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


models for video analysis. First, while their strong inductive
biases (e.g., local connectivity and translation equivariance)
are undoubtedly beneﬁcial on small training sets, they may
excessively limit the expressivity of the model in settings
where there is ample availability of data and “all” can be
learned from examples. Compared to CNNs, Transformers
impose less restrictive inductive biases. This broadens the
family of functions they can represent (Cordonnier et al.,
2020; Zhao et al., 2020), and renders them better suited
to modern big-data regimes where there is less need for
strong inductive priors. Second, while convolutional kernels
are speciﬁcally designed to capture short-range spatiotemporal information, they cannot model dependencies that
extend beyond the receptive ﬁeld. While deep stacks of
convolutions (Simonyan & Zisserman, 2015; Szegedy et al.,
2015; Carreira & Zisserman, 2017) naturally extend the
receptive ﬁeld, these strategies are inherently limited in capturing long-range dependencies by means of aggregation
of shorter-range information. Conversely, the self-attention
mechanism can be applied to capture both local as well as

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


of shorter-range information. Conversely, the self-attention
mechanism can be applied to capture both local as well as
global long-range dependencies by directly comparing feature activations at all space-time locations, much beyond the
receptive ﬁeld of traditional convolutional ﬁlters. Finally,arXiv:2102.05095v4  [cs.CV]  9 Jun 2021
Is Space-Time Attention All You Need for Video Understanding?
despite the advances in GPU hardware acceleration, training
deep CNNs remains very costly, especially when applied to
high-resolution and long videos. Recent work in the stillimage domain (Dosovitskiy et al., 2020; Carion et al., 2020;
Zhao et al., 2020) has demonstrated that Transformers enjoy
faster training and inference compared to CNNs, making it
possible to construct models with larger learning capacity
for the same computational budget.
Motivated by these observations, we propose a video architecture built exclusively on self-attention. We adapt the
image model “Vision Transformer” (ViT) (Dosovitskiy et al.,
2020) to video by extending the self-attention mechanism
from the image space to the space-time 3D volume. Our
proposed model, named “TimeSformer” (from Time-Space

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


from the image space to the space-time 3D volume. Our
proposed model, named “TimeSformer” (from Time-Space
Transformer), views the video as a sequence of patches extracted from the individual frames. As in ViT, each patch is
linearly mapped into an embedding and augmented with positional information. This makes it possible to interpret the
resulting sequence of vectors as token embeddings which
can be fed to a Transformer encoder, analogously to the
token features computed from words in NLP.
One downside of self-attention in standard Transformer is
that it requires computing a similarity measure for all pairs
of tokens. In our setting, this is computationally costly due
to the large number of patches in the video. To address these
challenges, we propose several scalable self-attention designs over the space-time volume and empirically evaluate
them over large-scale action classiﬁcation datasets. Among
the proposed schemes, we found that the best design is represented by a “divided attention” architecture which separately
applies temporal attention and spatial attention within each
block of the network. Compared to the established paradigm
of convolution-based video architecture, TimeSformer follows a radically different design. Yet, it achieves accuracy

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


block of the network. Compared to the established paradigm
of convolution-based video architecture, TimeSformer follows a radically different design. Yet, it achieves accuracy
comparable, and in some cases superior, to the state-of-theart in this ﬁeld. We also show that our model can be used
for long-range modeling of videos spanning many minutes.
2. Related Work
Our approach is inﬂuenced by recent works that use selfattention for image classiﬁcation, either in combination with
the convolution operator or even as a full replacement for it.
Within the former class, Non-Local Networks (Wang et al.,
2018b) employ a non-local mean that effectively generalizes
the self-attention function of Transformers (Vaswani et al.,
2017b). Bello et al. (Bello et al., 2019) propose a 2D selfattention mechanism that is competitive as a replacement of
2D convolution but gives even stronger results when used to
augment convolutional features with self-attention features.
Beyond image categorization, Relation Networks (Hu et al.,
2018) and DETR (Carion et al., 2020) use self-attention on

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Beyond image categorization, Relation Networks (Hu et al.,
2018) and DETR (Carion et al., 2020) use self-attention on
top of convolutional feature maps for object detection.Our method is more closely related to image networks leveraging self-attention as a substitute for convolution (Parmar
et al., 2018; Ramachandran et al., 2019; Cordonnier et al.,
2020; Zhao et al., 2020). Since these works use individual
pixels as queries, in order to maintain a manageable computational cost and a small memory consumption, they must
restrict the scope of self-attention to local neighborhoods or
use global self-attention on heavily downsized versions of
the image. Alternative strategies for scalability to full images include sparse key-value sampling (Child et al., 2019)
or constraining the self-attention to be calculated along the
spatial axes (Ho et al., 2019; Huang et al., 2019; Wang
et al., 2020b). A few of the self-attention operators considered in our experiments adopt similar sparse and axial
computation, although generalized to the spatiotemporal
volume. However, the efﬁciency of our approach stems
mainly from decomposing the video into a sequence of
frame-level patches and then feeding linear embeddings of

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


mainly from decomposing the video into a sequence of
frame-level patches and then feeding linear embeddings of
these patches as input token embeddings to a Transformer.
This strategy was recently introduced in Vision Transformers (ViT) (Dosovitskiy et al., 2020) which were shown to
deliver impressive performance on image categorization. In
this work, we build on the ViT design, and extend it to video
by proposing and empirically comparing several scalable
schemes for space-time self-attention over videos.
While Transformers have been recently used for video generation (Weissenborn et al., 2020), we are not aware of prior
video recognition architectures using self-attention as the
exclusive building block. However, we note that Transformers have been adopted on top of convolutional feature
maps for action localization and recognition (Girdhar et al.,
2019), video classiﬁcation (Wang et al., 2018b; Chen et al.,
2018b), and group activity recognition (Gavrilyuk et al.,
2020). We also note that there is a wide literature based on
the use of text Transformers combined with video CNNs
to address various video-language tasks, such as captioning (Zhou et al., 2018), question-answering (Yang et al.,

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


to address various video-language tasks, such as captioning (Zhou et al., 2018), question-answering (Yang et al.,
2020) and dialog (Le et al., 2019). Finally, multimodal
video-text transformers (Sun et al., 2019; Li et al., 2020a)
have also been trained or pretrained in unsupervised fashion
by adopting masked-token pretext tasks adapted from the
language domain (Devlin et al., 2018; Radford et al., 2018).
3. The TimeSformer Model
Input clip. The TimeSformer takes as input a clip X2
RHW3Fconsisting of FRGB frames of size HW
sampled from the original video.
Decomposition into patches. Following the ViT (Dosovitskiy et al., 2020), we decompose each frame into N
non-overlapping patches, each of size PP, such that
theNpatches span the entire frame, i.e., N=HW=P2.
We ﬂatten these patches into vectors x(p;t)2R3P2with
Is Space-Time Attention All You Need for Video Understanding?

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Space Attention (S)  Joint Space-Time      Attention (ST)Divided Space-Time     Attention (T+S)<latexit

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Axial Attention      (T+W+H)Sparse Local Global     Attention (L+G)MLPLocal Att.Global Att.MLPz(`)

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Figure 1. The video self-attention blocks that we investigate in this work. Each attention layer implements self-attention (Vaswani et al.,
2017b) on a speciﬁed spatiotemporal neighborhood of frame-level patches (see Figure 2 for a visualization of the neighborhoods). We use
residual connections to aggregate information from different attention layers within each block. A 1-hidden-layer MLP is applied at the
end of each block. The ﬁnal model is constructed by repeatedly stacking these blocks on top of each other.
p= 1;:::;N denoting spatial locations and t= 1;:::;F
depicting an index over frames.
Linear embedding. We linearly map each patch x(p;t)into
an embedding vector z(0)
(p;t)2RDby means of a learnable
matrixE2RD3P2:
z(0)
(p;t)=Ex(p;t)+epos
(p;t)(1)
where epos
(p;t)2RDrepresents a learnable positional embedding added to encode the spatiotemporal position of each
patch. The resulting sequence of embedding vectors z(0)
(p;t)

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


patch. The resulting sequence of embedding vectors z(0)
(p;t)
forp= 1;:::;N , andt= 1;:::;F represents the input to
the Transformer, and plays a role similar to the sequences of
embedded words that are fed to text Transformers in NLP.
As in the original BERT Transformer (Devlin et al., 2018),
we add in the ﬁrst position of the sequence a special learnable vector z(0)
(0;0)2RDrepresenting the embedding of the
classiﬁcation token.
Query-Key-Value computation. Our Transformer consists
ofLencoding blocks. At each block `, a query/key/value
vector is computed for each patch from the representation
z(`1)
(p;t)encoded by the preceding block:
q(`;a)
(p;t)=W(`;a)
QLN
z(`1)
(p;t)
2RDh(2)
k(`;a)
(p;t)=W(`;a)
KLN
z(`1)
(p;t)
2RDh(3)
v(`;a)

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


KLN
z(`1)
(p;t)
2RDh(3)
v(`;a)
(p;t)=W(`;a)
VLN
z(`1)
(p;t)
2RDh(4)where LN() denotes LayerNorm (Ba et al., 2016), a=
1;:::;Ais an index over multiple attention heads and A
denotes the total number of attention heads. The latent
dimensionality for each attention head is set to Dh=D=A.
Self-attention computation. Self-attention weights are
computed via dot-product. The self-attention weights
(`;a)
(p;t)2RNF+1for query patch (p;t)are given by:
(`;a)
(p;t)= SM0
@q(`;a)
(p;t)pDh>
"
k(`;a)
(0;0)n
k(`;a)
(p0;t0)o
p0=1;:::;N
t0=1;:::;F#1
A
(5)
where SMdenotes the softmax activation function. Note

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


p0=1;:::;N
t0=1;:::;F#1
A
(5)
where SMdenotes the softmax activation function. Note
that when attention is computed over one dimension only
(e.g., spatial-only or temporal-only), the computation is
signiﬁcantly reduced. For example, in the case of spatial
attention, only N+ 1 query-key comparisons are made,
using exclusively keys from the same frame as the query:
(`;a)space
(p;t)= SM0
@q(`;a)
(p;t)pDh>

k(`;a)
(0;0)n
k(`;a)
(p0;t)o
p0=1;:::;N1
A:
(6)
Encoding. The encoding z(`)
(p;t)at block`is obtained by
ﬁrst computing the weighted sum of value vectors using
self-attention coefﬁcients from each attention head:
Is Space-Time Attention All You Need for Video Understanding?
frame tframe t -

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Attention (S)   Joint Space-Time       Attention (ST)Divided Space-Time      Attention (T+S)  Sparse Local Global        Attention (L+G)  Axial Attention        (T+W+H)

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Figure 2. Visualization of the ﬁve space-time self-attention schemes studied in this work. Each video clip is viewed as a sequence of
frame-level patches with a size of 1616pixels. For illustration, we denote in blue the query patch and show in non-blue colors its
self-attention space-time neighborhood under each scheme. Patches without color are not used for the self-attention computation of the
blue patch. Multiple colors within a scheme denote attentions separately applied along different dimensions (e.g., space and time for
(T+S)) or over different neighborhoods (e.g., for (L+G)). Note that self-attention is computed for every single patch in the video clip, i.e.,
every patch serves as a query. We also note that although the attention pattern is shown for only two adjacent frames, it extends in the
same fashion to all frames of the clip.
s(`;a)
(p;t)=(`;a)
(p;t);(0;0)v(`;a)
(0;0)+NX
p0=1FX
t0=1(`;a)
(p;t);(p0;t0)v(`;a)
(p0;t0):
(7)

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


(p;t);(p0;t0)v(`;a)
(p0;t0):
(7)
Then, the concatenation of these vectors from all heads
is projected and passed through an MLP, using residual
connections after each operation:
z0(`)
(p;t)=WO2
664s(`;1)
(p;t)
...
s(`;A)
(p;t)3
775+z(`1)
(p;t)(8)
z(`)
(p;t)= MLP
LN
z0(`)
(p;t)
+z0(`)
(p;t): (9)
Classiﬁcation embedding. The ﬁnal clip embedding is
obtained from the ﬁnal block for the classiﬁcation token:
y= LN
z(L)
(0;0)
2RD: (10)
On top of this representation we append a 1-hidden-layer
MLP, which is used to predict the ﬁnal video classes.
Space-Time Self-Attention Models. We can reduce the

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


MLP, which is used to predict the ﬁnal video classes.
Space-Time Self-Attention Models. We can reduce the
computational cost by replacing the spatiotemporal attention of Eq. 5 with spatial attention within each frame only
(Eq. 6). However, such a model neglects to capture temporaldependencies across frames. As shown in our experiments,
this approach leads to degraded classiﬁcation accuracy compared to full spatiotemporal attention, especially on benchmarks where strong temporal modeling is necessary.
We propose a more efﬁcient architecture for spatiotemporal
attention, named “Divided Space-Time Attention” (denoted
with T+S), where temporal attention and spatial attention
are separately applied one after the other. This architecture
is compared to that of Space and Joint Space-Time attention
in Fig. 1. A visualization of the different attention models
on a video example is given in Fig. 2. For Divided Attention,
within each block `, we ﬁrst compute temporal attention by
comparing each patch (p;t)with all the patches at the same
spatial location in the other frames:
(`;a)time
(p;t)= SM0
@q(`;a)
(p;t)pDh>

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


(`;a)time
(p;t)= SM0
@q(`;a)
(p;t)pDh>

k(`;a)
(0;0)n
k(`;a)
(p;t0)o
t0=1;:::;F1
A:
(11)
The encoding z0(`)time
(p;t)resulting from the application of
Eq. 8 using temporal attention is then fed back for spatial
attention computation instead of being passed to the MLP. In
other words, new key/query/value vectors are obtained from
z0(`)time
(p;t)and spatial attention is then computed using Eq. 6.
Finally, the resulting vector z0(`)space
(p;t)is passed to the MLP
of Eq. 9 to compute the ﬁnal encoding z(`)
(p;t)of the patch at
block`. For the model of divided attention, we learn distinct query/key/value matrices fW(`;a)
Qtime;W(`;a)
Ktime;W(`;a)
Vtimeg
Is Space-Time Attention All You Need for Video Understanding?

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Qtime;W(`;a)
Ktime;W(`;a)
Vtimeg
Is Space-Time Attention All You Need for Video Understanding?
Attention Params K400 SSv2
Space 85.9M 76.9 36.6
Joint Space-Time 85.9M 77.4 58.5
Divided Space-Time 121.4M 78.0 59.5
Sparse Local Global 121.4M 75.9 56.3
Axial 156.8M 73.5 56.2
Table 1. Video-level accuracy for different space-time attention
schemes in TimeSformer. We evaluate the models on the validation sets of Kinetics-400 (K400), and Something-Something-V2
(SSv2). We observe that divided space-time attention achieves the
best results on both datasets.
andfW(`;a)
Qspace;W(`;a)
Kspace;W(`;a)
Vspacegover the time and space
dimensions. Note that compared to the (NF+ 1) comparisons per patch needed by the joint spatiotemporal attention
model of Eq. 5, Divided Attention performs only (N+F+2)
comparisons per patch. Our experiments demonstrate that

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


model of Eq. 5, Divided Attention performs only (N+F+2)
comparisons per patch. Our experiments demonstrate that
this space-time factorization is not only more efﬁcient but it
also leads to improved classiﬁcation accuracy.
We have also experimented with a “Sparse Local Global”
(L+G) and an “Axial” (T+W+H) attention models. Their
architectures are illustrated in Fig. 1, while Fig. 2 shows
the patches considered for attention by these models. For
each patch (p;t), (L+G) ﬁrst computes a local attention by
considering the neighboring FH=2W=2patches and
then calculates a sparse global attention over the entire clip
using a stride of 2 patches along the temporal dimension and
also the two spatial dimensions. Thus, it can be viewed as a
faster approximation of full spatiotemporal attention using a
local-global decomposition and a sparsity pattern, similar to
that used in (Child et al., 2019). Finally, “Axial” attention
decomposes the attention computation in three distinct steps:
over time, width and height. A decomposed attention over
the two spatial axes of the image was proposed in (Ho

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


decomposes the attention computation in three distinct steps:
over time, width and height. A decomposed attention over
the two spatial axes of the image was proposed in (Ho
et al., 2019; Huang et al., 2019; Wang et al., 2020b) and
our (T+W+H) adds a third dimension (time) for the case
of video. All these models are implemented by learning
distinct query/key/value matrices for each attention step.
4. Experiments
We evaluate TimeSformer on four popular action recognition datasets: Kinetics-400 (Carreira & Zisserman, 2017),
Kinetics-600 (Carreira et al., 2018), Something-SomethingV2 (Goyal et al., 2017b), and Diving-48 (Li et al., 2018). We
adopt the “Base” ViT architecture (Dosovitskiy et al., 2020)
pretrained on either ImageNet-1K or ImageNet-21K (Deng
et al., 2009), as speciﬁed for each experiment. Unless differently indicated, we use clips of size 8224224, with
frames sampled at a rate of 1=32. The patch size is 1616
pixels. During inference, unless otherwise noted, we sample a single temporal clip in the middle of the video. We

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


pixels. During inference, unless otherwise noted, we sample a single temporal clip in the middle of the video. We
use3spatial crops (top-left, center, bottom-right) from the
temporal clip and obtain the ﬁnal prediction by averaging
the scores for these 3crops.
8 32 64 96# of Input frames0510TFLOPsJoint Space-TimeDivided Space-Time224 336 448 560Spatial Crop (Px)0123TFLOPsJoint Space-TimeDivided Space-Time
Out of memory
Out of memoryFigure 3. We compare the video classiﬁcation cost (in TFLOPs) of
Joint Space-Time versus Divided Space-Time attention. We plot
the number of TFLOPs as a function of spatial crop size in pixels
(left), and the number of input frames (right). As we increase the
spatial resolution (left), or the video length (right), our proposed divided space-time attention leads to dramatic computational savings
compared to the scheme of joint space-time attention.
4.1. Analysis of Self-Attention Schemes
For this ﬁrst set of experiments we start from a ViT pretrained on ImageNet-21K. In Table 1, we present the results

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


For this ﬁrst set of experiments we start from a ViT pretrained on ImageNet-21K. In Table 1, we present the results
obtained with TimeSformer for the ﬁve proposed space-time
attention schemes on Kinetics-400 (K400) and SomethingSomething-V2 (SSv2). First, we note that TimeSformer
with space-only attention (S) performs well on K400. This
is an interesting ﬁnding. Indeed, prior work (Sevilla-Lara
et al., 2021) has shown that on K400, spatial cues are more
important than temporal information in order to achieve
strong accuracy. Here, we show that it is possible to obtain
solid accuracy on K400 without any temporal modeling.
Note, however, that space-only attention performs poorly on
SSv2. This stresses the importance of temporal modeling
on this latter dataset.
Furthermore, we observe that divided space-time attention
achieves the best accuracy on both K400 and SSv2. This
makes sense because compared to joint space-time attention,
divided space-time attention has a larger learning capacity
(see Table 1) as it contains distinct learning parameters for
temporal attention and spatial attention.
In Figure 3, we also compare the computational cost of joint
space-time versus divided space-time attention when using

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


temporal attention and spatial attention.
In Figure 3, we also compare the computational cost of joint
space-time versus divided space-time attention when using
higher spatial resolution (left) and longer (right) videos. We
note that the scheme of divided space-time scales gracefully
under both of these settings. In contrast, the scheme of
joint space-time attention leads to a dramatically higher cost
when resolution or video length is increased. In practice,
joint space-time attention causes a GPU memory overﬂow
once the spatial frame resolution reaches 448pixels, or once
the number of frames is increased to 32and thus it is effectively not applicable to large frames or long videos. Thus,
despite a larger number of parameters, divided space-time
attention is more efﬁcient than joint space-time attention
when operating on higher spatial resolution, or longer videos.
Thus, for all subsequent experiments we use a TimeSformer
constructed with divided space-time self-attention blocks.
Is Space-Time Attention All You Need for Video Understanding?
Model Pretrain K400 Training
Time (hours)K400
Acc.Inference
TFLOPsParams
I3D 8x8 R50 ImageNet-1K 444 71.0 1.11 28.0M

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Acc.Inference
TFLOPsParams
I3D 8x8 R50 ImageNet-1K 444 71.0 1.11 28.0M
I3D 8x8 R50 ImageNet-1K 1440 73.4 1.11 28.0M
SlowFast R50 ImageNet-1K 448 70.0 1.97 34.6M
SlowFast R50 ImageNet-1K 3840 75.6 1.97 34.6M
SlowFast R50 N/A 6336 76.4 1.97 34.6M
TimeSformer ImageNet-1K 416 75.8 0.59 121.4M
TimeSformer ImageNet-21K 416 78.0 0.59 121.4M
Table 2. Comparing TimeSformer to SlowFast and I3D. We observe that TimeSformer has lower inference cost despite having
a larger number of parameters. Furthermore, the cost of training
TimeSformer on video data is much lower compared to SlowFast
and I3D, even when all models are pretrained on ImageNet-1K.
4.2. Comparison to 3D CNNs
In this subsection we perform an empirical study aimed
at understanding the distinguishing properties of TimeSformer compared to 3D convolutional architectures, which
have been the prominent approach to video understanding

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


In this subsection we perform an empirical study aimed
at understanding the distinguishing properties of TimeSformer compared to 3D convolutional architectures, which
have been the prominent approach to video understanding
in recent years. We focus our comparison on two 3D CNN
models: 1) SlowFast (Feichtenhofer et al., 2019b), which is
the state-of-the-art in video classiﬁcation, and 2) I3D (Carreira & Zisserman, 2017), which has been shown to beneﬁt
from image-based pretraining, similarly to our own model.
We present quantitative comparisons to these two networks
in Table 2 and highlight key observations below.
Model Capacity. From Table 2, we ﬁrst observe that although TimeSformer has a large learning capacity (the number of parameters is 121:4M), it has low inference cost
(0:59in TFLOPs). In contrast, SlowFast 8x8 R50 has a
larger inference cost ( 1:97TFLOPs) despite containing only
34:6Mparameters. Similarly, I3D 8x8 R50 also has a larger
inference cost ( 1:11TFLOPs) despite containing fewer parameters ( 28:0M). This suggests that TimeSformer is better

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


inference cost ( 1:11TFLOPs) despite containing fewer parameters ( 28:0M). This suggests that TimeSformer is better
suited for settings that involve large-scale learning. In contrast, the large computational cost of modern 3D CNNs
makes it difﬁcult to further increase their model capacity
while also maintaining efﬁciency.
Video Training Time. One signiﬁcant advantage of ImageNet pretraining is that it enables very efﬁcient training of
TimeSformer on video data. Conversely, state-of-the-art 3D
CNNs are much more expensive to train even if pretrained
on image datasets. In Table 2, we compare the video training time on Kinetics-400 (in Tesla V100 GPU hours) of
TimeSformer to that of SlowFast and I3D. Starting from a
ResNet50 pretrained on ImageNet-1K, SlowFast 88R50
requires 3 840 Tesla V100 GPU hours in order to reach an
accuracy of 75:6%on Kinetics-400. Training I3D, under
similar settings, requires 1 440 Tesla V100 GPU hours for a
73:4%accuracy. In contrast, TimeSformer, also pretrained
on ImageNet-1K, only requires 416Tesla V100 GPU hours

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


73:4%accuracy. In contrast, TimeSformer, also pretrained
on ImageNet-1K, only requires 416Tesla V100 GPU hours
to achieve a higher 75:8%accuracy (see Table 2). Furthermore, if we constrain SlowFast to be trained under a
somewhat similar computational budget as TimeSformerMethod Pretraining K400 SSv2
TimeSformer ImageNet-1K 75.8 59.5
TimeSformer ImageNet-21K 78.0 59.5
TimeSformer-HR ImageNet-1K 77.8 62.2
TimeSformer-HR ImageNet-21K 79.7 62.5
TimeSformer-L ImageNet-1K 78.1 62.4
TimeSformer-L ImageNet-21K 80.7 62.3
Table 3. Comparing the effectiveness of ImageNet-1K and
ImageNet-21K pretraining on Kinetics-400 (K400) and SomethingSomething-V2 (SSv2). On K400, ImageNet-21K pretraining leads
consistently to a better performance compared to ImageNet-1K pretraining. On SSv2, ImageNet-1K and ImageNet-21K pretrainings
lead to similar accuracy.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


lead to similar accuracy.
(i.e.,448GPU hours), its accuracy drops to 70:0%. Similarly, training I3D using a similar computational budget (i.e.,
444GPU hours) leads to a lower accuracy of 71:0%. This
highlights the fact that some of the latest 3D CNNs (Feichtenhofer et al., 2019b; Feichtenhofer, 2020) require a very
long optimization schedule to achieve good performance
(even when using ImageNet pretraining). In contrast, TimeSformer provides a more efﬁcient alternative to labs that do
not have access to hundreds of GPUs.
The Importance of Pretraining. Due to a large number
of parameters, training our model from scratch is difﬁcult.
Thus, before training TimeSformer on video data, we initialize it with weights learned from ImageNet. In contrast,
SlowFast can be learned on video data from scratch although
at the expense of a very high training cost (see Table 2). We
also attempted to train TimeSformer on Kinetics-400 directly, without any ImageNet pretraining. By using a longer
training schedule and more data augmentations, we found
it possible to train the model from scratch, albeit to a much
lower video level accuracy of 64:8%. Thus, based on these

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


training schedule and more data augmentations, we found
it possible to train the model from scratch, albeit to a much
lower video level accuracy of 64:8%. Thus, based on these
results, for all subsequent studies we continued to use ImageNet for pretraining (Deng et al., 2009)
In Table 3 we study the beneﬁts of ImageNet-1K vs
ImageNet-21K pretraining on K400 and SSv2. For these
experiments, we use three variants of our model: (1) TimeSformer , which is the default version of our model operating
on8224224video clips, (2) TimeSformer-HR , a high
spatial resolution variant that operates on 16448448
video clips, and lastly (3) TimeSformer-L , a long-range
conﬁguration of our model that operates on 96224224
video clips with frames sampled at a rate of 1=4.
Based on the results in Table 3, we observe that ImageNet21K pretraining is beneﬁcial for K400, where it leads to
a consistently higher accuracy compared to ImageNet-1K
pretraining. On the other hand, on SSv2, we observe that

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


a consistently higher accuracy compared to ImageNet-1K
pretraining. On the other hand, on SSv2, we observe that
ImageNet-1K and ImageNet-21K pretrainings lead to similar accuracy. This makes sense as SSv2 requires complex
spatiotemporal reasoning, whereas K400 is biased more towards spatial scene information, and thus, it beneﬁts more
from the features learned on the larger pretraining dataset.
Is Space-Time Attention All You Need for Video Understanding?
60K 120K 180K 240K# of Training Videos60657075AccuracyKinetics
TimeSformerSlowFastI3D42K 85K 127K 170K# of Training Videos354045505560AccuracySomething-Something-V2
TimeSformerSlowFastI3D
Figure 4. Accuracy on Kinetics-400 (K400), and SomethingSomething-V2 (SSv2) as a function of the number of training
videos. On K400, TimeSformer performs best in all cases. On
SSv2, which requires more complex temporal reasoning, TimeSformer outperforms the other models only when using enough
training videos. All models are pretrained on ImageNet-1K.
The Impact of Video-Data Scale. To understand the
effects of video-data scale on performance, we trained

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


training videos. All models are pretrained on ImageNet-1K.
The Impact of Video-Data Scale. To understand the
effects of video-data scale on performance, we trained
TimeSformer on different subsets of K400 and SSv2:
f25%;50%;75%;100%gof the full datasets. We show
these results in Figure 4, where we also compare our method
with SlowFast R50 (Feichtenhofer et al., 2019b), and I3D
R50 (Carreira & Zisserman, 2017) trained on the same subsets and using the same pretraining. Since we do not have
access to a ResNet pretrained on ImageNet-21K, we use
ImageNet-1K pretraining for all 3architectures.
The results of Figure 4 show that, on K400, TimeSformer
outperforms the other models for all training subsets. However, we observe a different trend on SSv2, where TimeSformer is the strongest model only when trained on 75% or
100% of the full data. This may be explained by the fact that
compared to K400, SSv2 requires learning more complex
temporal patterns, and thus more examples are needed by
TimeSformer to learn effectively those patterns.
4.3. Varying the Number of Tokens

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


temporal patterns, and thus more examples are needed by
TimeSformer to learn effectively those patterns.
4.3. Varying the Number of Tokens
The scalability of our model allows it to operate at higher
spatial resolution and on longer videos compared to most 3D
CNNs. We note that both of these aspects affect the length of
the sequence of tokens fed to the Transformer. Speciﬁcally,
increasing the spatial resolution results in a higher number
of patches (N) per frame. The number of input tokens is
also increased when using more frames. To investigate the
beneﬁts, we conduct an empirical study where we separately
increase the number of tokens along each of these two axes.
We report the ﬁndings in Figure 5. We see that increasing
the spatial resolution (up to a certain point) leads to a boost
in performance. Similarly, we observe that increasing the
length of the input clip leads to consistent accuracy gains.
Due to GPU memory constraints, we are not able to test
our model on clips longer than 96frames. Still, we would
like to point out that using clips of 96frames is a signiﬁcant
departure from current convolutional models, which are
typically limited to processing inputs of 832frames.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


departure from current convolutional models, which are
typically limited to processing inputs of 832frames.
8 32 64 96Number of Input Frames65707580Clip Accuracy224 336 448 560Spatial Crop Side (Px)65707580Clip AccuracyFigure 5. Clip-level accuracy on Kinetics-400 as a function of spatial crop size in pixels (left), and the number of input frames (right).
Positional Embedding K400 SSv2
None 75.4 45.8
Space-only 77.8 52.5
Space-Time 78.0 59.5
Table 4. Ablation on positional embeddings. The version of TimeSformer using space-time positional embeddings yields the highest
accuracy on both Kinetics-400 and SSv2.
4.4. The Importance of Positional Embeddings
To investigate the importance of our learned spatiotemporal positional embeddings, we also conduct experiments
with a few variants of TimeSformer that use: (1) no positional embedding, (2) space-only positional embedding,
and (3) space-time positional embedding. We report these
results in Table 4. Based on these results, we observe
that the variant of our model that uses space-time positional embeddings produces the best accuracy on both

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


results in Table 4. Based on these results, we observe
that the variant of our model that uses space-time positional embeddings produces the best accuracy on both
Kinetics-400, and Something-Something-V2. Interestingly,
we also observe that using space-only positional embeddings leads to solid results on Kinetics-400, but much worse
results on Something-Something-V2. This makes sense as
Kinetics-400 is more spatially biased, whereas SomethingSomething-V2 requires complex temporal reasoning.
4.5. Comparison to the State-of-the-Art
Kinetics-400 & Kinetics-600. In Table 5 we present our
results on the validation set of K400. For these experiments,
we use TimeSformer pretrained on ImageNet-21K. In addition to the accuracy metrics, we also include inference
cost, given in TFLOPs. We note that whereas most previous
Method Top-1 Top-5 TFLOPs
R(2+1)D (Tran et al., 2018) 72.0 90.0 17.5
bLVNet (Fan et al., 2019) 73.5 91.2 0.84
TSM (Lin et al., 2019) 74.7 N/A N/A
S3D-G (Xie et al., 2018) 74.7 93.4 N/A

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


S3D-G (Xie et al., 2018) 74.7 93.4 N/A
Oct-I3D+NL (Chen et al., 2019) 75.7 N/A 0.84
D3D (Stroud et al., 2020) 75.9 N/A N/A
I3D+NL (Wang et al., 2018b) 77.7 93.3 10.8
ip-CSN-152 (Tran et al., 2019) 77.8 92.8 3.2
CorrNet (Wang et al., 2020a) 79.2 N/A 6.7
LGD-3D-101 (Qiu et al., 2019) 79.4 94.4 N/A
SlowFast (Feichtenhofer et al., 2019b) 79.8 93.9 7.0
X3D-XXL (Feichtenhofer, 2020) 80.4 94.6 5.8
TimeSformer 78.0 93.7 0.59
TimeSformer-HR 79.7 94.4 5.11
TimeSformer-L 80.7 94.7 7.14
Table 5. Video-level accuracy on Kinetics-400.
Is Space-Time Attention All You Need for Video Understanding?
Method Top-1 Top-5

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Table 5. Video-level accuracy on Kinetics-400.
Is Space-Time Attention All You Need for Video Understanding?
Method Top-1 Top-5
I3D-R50+Cell (Wang et al., 2020c) 79.8 94.4
LGD-3D-101 (Qiu et al., 2019) 81.5 95.6
SlowFast (Feichtenhofer et al., 2019b) 81.8 95.1
X3D-XL (Feichtenhofer, 2020) 81.9 95.5
TimeSformer 79.1 94.4
TimeSformer-HR 81.8 95.8
TimeSformer-L 82.2 95.6
Table 6. Video-level accuracy on Kinetics-600.
1 3 5 10
# of Testing Clips65707580Accuracy
TimeSformer-L
TimeSformer
SlowFast-R101+NL
X3D-XL
Figure 6. Video-level accuracy on Kinetics-400 vs the number of
temporal clips used during inference. TimeSformer-L achieves
excellent accuracy using a small number of clips, which leads to
strong performance at low inference cost.
methods use 10temporal clips with 3spatial crops (for a total of 30space-time views) during inference, TimeSformer

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


strong performance at low inference cost.
methods use 10temporal clips with 3spatial crops (for a total of 30space-time views) during inference, TimeSformer
achieves solid accuracy with only 3views (3 spatial crops),
which reduces the inference cost. Our long-range variant,
TimeSformer-L achieves a top-1 accuracy of 80:7%. Furthermore, our default TimeSformer has the lowest inference
cost among recent state-of-the-art models. Yet, it still provides a solid accuracy of 78:0%, outperforming many more
costly models.
We also measured the actual inference runtime on 20Kvalidation videos of Kinetics-400 (using 8Tesla V100 GPUs).
Whereas SlowFast takes 14:88hours to complete the inference, TimeSformer, TimeSformer-HR, and TimeSformer-L
take36minutes, 1:06hours and 2:6hours, respectively.
Thus, even though SlowFast and TimeSformer-L have comparable cost in terms of TFLOPs, in practice the runtimes
of all our versions of TimeSformer are much lower.
In Table 6, we also present our results on Kinetics-600.
Just like on Kinetics-400, we observe that TimeSformer
performs well on this benchmark, outperforming all prior
methods.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Just like on Kinetics-400, we observe that TimeSformer
performs well on this benchmark, outperforming all prior
methods.
Finally, in Figure 6, we study the effect of using multiple temporal clips during inference (each with a single
spatial crop). We plot accuracy using K2f1;3;5;10g
temporal clips for testing. We compare our model against
X3D (Feichtenhofer, 2020), and SlowFast (Feichtenhofer
et al., 2019b). X3D and SlowFast require multiple ( 5)
clips to approach their top accuracy. Conversely, our longrange variant, TimeSformer-L, does not require multiple
clips to achieve its best performance, since it is able to spanMethod SSv2 Diving-48
SlowFast (Feichtenhofer et al., 2019b) 61.7 77.6
TSM (Lin et al., 2019) 63.4 N/A
STM (Jiang et al., 2019) 64.2 N/A
MSNet (Kwon et al., 2020) 64.7 N/A
TEA (Li et al., 2020b) 65.1 N/A
bLVNet (Fan et al., 2019) 65.2 N/A
TimeSformer 59.5 74.9

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


bLVNet (Fan et al., 2019) 65.2 N/A
TimeSformer 59.5 74.9
TimeSformer-HR 62.2 78.0
TimeSformer-L 62.4 81.0
Table 7. Video-level accuracy on Something-Something-V2 and
Diving-48.Due to an issue with Diving-48 labels used in previously published results, we only compare our method with a
reproduced SlowFast 168R101 model. All models are pretained on ImageNet-1K.
about 12seconds of a Kinetics video with a single clip.
Something-Something-V2 & Diving-48. In Table 7, we
also validate our model on SSv2 and Diving-48. Since
ImageNet-21K pretraining does not improve accuracy on
SSv2 (see Table 3), in this case, we use TimeSformer pretrained on ImageNet-1K. This also allows us to apply the
same pretraining to all other models in this comparison,
using a ResNet pretrained on ImageNet-1K. Our results suggest that TimeSformer achieves lower accuracy than the best
models on this dataset. However, considering that our model
uses a completely different design, we take these results as
suggesting that TimesSformer is a promising approach even

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


models on this dataset. However, considering that our model
uses a completely different design, we take these results as
suggesting that TimesSformer is a promising approach even
for challenging temporally-heavy datasets, such as SSv2.
In Table 7, we also present our method on another
“temporally-heavy” dataset, Diving-48. Due to a recently
discovered issue with a previous version of Diving-48 labels,
here, we only compare our method with a reproduced SlowFast168R101 model. Our results show that TimeSformer
outperforms SlowFast by a substantial margin.
4.6. Long-Term Video Modeling
Lastly, we evaluate TimeSformer on the task of long-term
video modeling using HowTo100M (Miech et al., 2019).
HowTo100M is an instructional video dataset that contains
around 1M instructional Web videos showing humans performing over 23K different tasks, such as cooking, repairing,
making arts, etc. The average duration of these videos is
around 7minutes, which is orders of magnitude longer than
the duration of videos in standard action recognition benchmarks. Each HowTo100M video has a label indicating the
task demonstrated in the video (one out of the 23K classes),
which can be used for supervised training. Thus, it is a good

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


task demonstrated in the video (one out of the 23K classes),
which can be used for supervised training. Thus, it is a good
benchmark to assess the ability of a model to recognize
activities exhibited over very long temporal extents.
For this evaluation, we consider only categories that have
at least 100 video examples. This gives a subset of
HowTo100M corresponding to 120Kvideos spanning 1059
task categories. We randomly partition this collection into
85Ktraining videos and 35Ktesting videos.
Is Space-Time Attention All You Need for Video Understanding?
Method # Input
FramesSingle Clip
Coverage# Test
ClipsTop-1
Acc
SlowFast 8 8.5s 48 48.2
SlowFast 32 34.1s 12 50.8
SlowFast 64 68.3s 6 51.5
SlowFast 96 102.4s 4 51.2
TimeSformer 8 8.5s 48 56.8
TimeSformer 32 34.1s 12 61.2
TimeSformer 64 68.3s 6 62.2
TimeSformer 96 102.4s 4 62.6
Table 8. Long-term task classiﬁcation on HowTo100M. Given a
video spanning several minutes, the goal is to predict the long-term
task demonstrated in the video (e.g., cooking breakfast, cleaning

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


video spanning several minutes, the goal is to predict the long-term
task demonstrated in the video (e.g., cooking breakfast, cleaning
house, etc). We evaluate a few variants of SlowFast and TimeSformer on this task. “Single Clip Coverage” denotes the number
of seconds spanned by a single clip. “# Test Clips” is the average
number of clips needed to cover the entire video during inference.
All models in this comparison are pretrained on Kinetics-400.
We present our results in Table 8. As our baselines, we use
four variants of SlowFast R101, all operating on video clips
sampled at a frame rate of 1=32but having varying number
of frames: 8;32;64and96. We use the same four conﬁgurations for TimeSformer, starting from a ViT pretrained
on ImageNet-21K. All models in this comparison are pretrained on Kinetics-400 before ﬁnetuning on HowTo100M.
During inference, for each method, we sample as many
non-overlapping temporal clips as needed to cover the full
temporal extent of a video, e.g., if a single clip spans 8:5
seconds, we would sample 48test clips to cover a video of
410seconds. Video-level classiﬁcation is done by averaging

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


seconds, we would sample 48test clips to cover a video of
410seconds. Video-level classiﬁcation is done by averaging
the clip predictions.
From the results in Table 8 we ﬁrst note that, for the same
single clip coverage, TimeSformer outperforms the corresponding SlowFast by a large margin of 811%. We also
observe that longer-range TimeSformers do better, i.e., our
longest-range variant achieves the best video-level classiﬁcation accuracy. These results suggest that our model is highly
suitable for tasks that require long-term video modeling.
We also experimented with ﬁnetuning TimeSformer directly
from a ViT pretrained on ImageNet-1K and ImageNet21K (skipping the Kinetics-400 training). We report that
when pretrained only on ImageNet-1K, our model achieves
top-1 accuracies of 52:8;58:4;59:2;59:4for8;32;64;96
frame inputs, respectively. When considering ImagNet21K pretraining, TimeSformer produces top-1 accuracies of
56:0;59:2;60:2;62:1for8;32;64;96frame inputs, respectively. These results demonstrate that our model can effectively exploit long-range temporal dependencies regardless

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


of the pretraining dataset that we use.
4.7. Additional Ablations
Smaller & Larger Transformers. In addition to the “Base”
ViT model (Dosovitskiy et al., 2020), we also experimented
with the “Large” ViT. We report that this yielded results 1%
Figure 7. Visualization of space-time attention from the output
token to the input space on Something-Something-V2. Our model
learns to focus on the relevant parts in the video in order to perform
spatiotemporal reasoning.
ViT-50 -40 -30 -20 -10 0 10 20 30 40 50-50-40-30-20-1001020304050
-60 -40 -20 0 20 40-40-30-20-1001020304050       TimeSformer w/ Divided            Space-Time Attention-60 -40 -20 0 20 40 60-60-40-200204060    TimeSformer  w/ Space Attention
Figure 8. Feature visualization with t-SNE (van der Maaten & Hinton, 2008) on Something-Something-V2. Each video is visualized
as a point. Videos belonging to the same action category have the
same color. The TimeSformer with divided space-time attention

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners