as a point. Videos belonging to the same action category have the
same color. The TimeSformer with divided space-time attention
learns semantically more separable features than the TimeSformer
with space-only attention or ViT (Dosovitskiy et al., 2020).
worse on both Kinetics-400, and Something-Something-V2.
Given that our “Base” model already has 121Mparameters,
we suspect that the current datasets are not big enough to
justify a further increase in model capacity. We also tried
the “Small” ViT variant, which produced accuracies about
5%worse than our default “Base” ViT model.
Larger Patch Size. We also experimented with a different
patch size, i.e., P= 32 . We report that this variant of our
model produced results about 3%worse than our default
variant using P= 16 . We conjecture that the performance
decrease with P= 32 is due to the reduced spatial granularity. We did not train any models with Pvalues lower than
16 as those models have a much higher computational cost.
The Order of Space and Time Self-Attention. Our proposed “Divided Space-Time Attention” scheme applies temporal attention and spatial attention one after the other. Here,
we investigate whether reversing the order of time-space

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


we investigate whether reversing the order of time-space
attention (i.e., applying spatial attention ﬁrst, then temporal) has an impact on our results. We report that applying spatial attention ﬁrst, followed by temporal attention
leads to a 0:5%drop in accuracy on both Kinetics-400, and
Something-Something-V2. We also tried a parallel spacetime self-attention. We report that it produces 0:4%lower
accuracy compared to our adopted “Divided Space-Time
Attention” scheme.
Is Space-Time Attention All You Need for Video Understanding?
4.8. Qualitative Results
Visualizing Learned Space-Time Attention. In Figure 7,
we present space-time attention visualizations obtained by
applying TimeSformer on Something-Something-V2 videos.
To visualize the learned attention, we use the Attention
Rollout scheme presented in (Abnar & Zuidema, 2020).
Our results suggest that TimeSformer learns to attend to the
relevant regions in the video in order to perform complex
spatiotemporal reasoning. For example, we can observe that
the model focuses on the conﬁguration of the hand when
visible and the object-only when not visible.
Visualizing Learned Feature Embeddings. In Figure 8,

Summary:
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.

Gedas Bertasius
Heng Wang
Lorenzo Torresani

Accepted to ICML 2021

Quantifying Attention Flow in Transformers

Layer Normalization

Lei Jimmy Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Attention Augmented Convolutional Networks

Bello, I., Zoph, B., Le, Q., Vaswani, A., Shlens, J.

Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation

Language Models are Few-Shot Learners


the model focuses on the conﬁguration of the hand when
visible and the object-only when not visible.
Visualizing Learned Feature Embeddings. In Figure 8,
we also visualize the features learned by TimeSformer on
Something-Something-V2. The visualization is done using
t-SNE (van der Maaten & Hinton, 2008) where each point
represents a single video, and different colors depict different action categories. Based on this illustration, we observe
that TimeSformer with divided space-time attention learns
semantically more separable features than the TimeSformer
with space-only attention or ViT (Dosovitskiy et al., 2020).
5. Conclusion
In this work, we introduced TimeSformer, a fundamentally
different approach to video modeling compared to the established paradigm of convolution-based video networks.
We showed that it is possible to design an effective, and
scalable video architecture built exclusively on space-time
self-attention. Our method (1) is conceptually simple, (2)
achieves state-of-the-art results on major action recognition
benchmarks, (3) has low training and inference cost, and (4)
can be applied to clips of over one minute, thus enabling

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


benchmarks, (3) has low training and inference cost, and (4)
can be applied to clips of over one minute, thus enabling
long-term video modeling. In the future, we plan to extend our method to other video analysis tasks such as action
localization, video captioning and question-answering.
Appendix
A. Implementation Details
Our TimeSformer implementation is built using PySlowFast (Fan et al., 2020) and pytorch-image-models (Wightman, 2019) packages. Below, we describe speciﬁc implementation details regarding the training and inference procedures of our model.
Training. We train our model for 15epochs with an initial
learning rate of 0:005, which is divided by 10at epochs
11;and14. During training, we ﬁrst resize the shorter
side of the video to a random value in [256;320]. We then
randomly sample a 224224crop from the resized video.
For our high-resolution model, TimeSformer-HR, we resize
the shorter side of the video to a random value in [448;512],
and then randomly sample a 448448crop. We randomlysample clips from the full-length videos with a frame rate
of1=32. The batch size is set to 16. We train all our models

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


of1=32. The batch size is set to 16. We train all our models
using synchronized SGD across 32GPUs. The momentum
is set to 0:9, while the weight decay is set to 0:0001 .
Unless otherwise noted, in our experiments we use the
“Base” ViT model (Dosovitskiy et al., 2020). Temporal and
spatial attention layers in each block are initialized with the
same weights, which are obtained from the corresponding
attention layer in ViT.
Inference. As discussed in the main draft, during inference
we sample a single temporal clip in the middle of the video.
We scale the shorter spatial side of a video to 224pixels (or
448for TimeSformer-HR) and take 3crops of size 224224
(448448for TimeSformer-HR) to cover a larger spatial
extent within the clip. The ﬁnal prediction is obtained by
averaging the softmax scores of these 3predictions.
Other models in our comparison. To train I3D (Carreira
& Zisserman, 2017), and SlowFast (Feichtenhofer et al.,
2019b), we use the training protocols that were used in the
original papers. For I3D, we initialize it with a 2D ImageNet

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


2019b), we use the training protocols that were used in the
original papers. For I3D, we initialize it with a 2D ImageNet
CNN, and then train it for 118epochs with a base learning
rate of 0:01, which is divided by 10at epochs 44and88.
We use synchronized SGD across 32GPUs following the
linear scaling recipe of Goyal et al. (2017a). We set the
momentum to 0:9, and weight decay to 0:0001 . The batch
size is set to 64. For the SlowFast model, when initialized
from ImageNet weights, we use this same exact training
protocol. When training SlowFast from scratch, we use the
training protocol described by the authors (Feichtenhofer
et al., 2019b). More speciﬁcally, in that case, the training
is done for 196epochs with a cosine learning rate schedule,
and the initial learning rate is set to 0:1. We use a linear
warm-up for the ﬁrst 34epochs starting with a learning rate
of0:01. A dropout of 0:5is used before the ﬁnal classiﬁcation layer. The momentum is set to 0:9, the weight decay is

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


0:0001 , and the batch size is set to 64. Just as before, we
adopt the linear scaling recipe (Goyal et al., 2017a).
Datasets. Kinetics-400 (Carreira & Zisserman, 2017) consists of 240Ktraining videos and 20Kvalidation videos
that span 400human action categories. Kinetics-600 (Carreira et al., 2018) has 392Ktraining videos and 30Kvalidation videos spanning 600action categories. SomethingSomething-V2 (Goyal et al., 2017b) contains 170Ktraining
videos and 25Kvalidation videos that span 174action categories. Lastly, Diving-48 (Li et al., 2018) has 16Ktraining
videos and 3Ktesting videos spanning 48ﬁne-grained diving categories. For all of these datasets, we use standard
classiﬁcation accuracy as our main performance metric.
Is Space-Time Attention All You Need for Video Understanding?
References
Abnar, S. and Zuidema, W. Quantifying attention ﬂow in
transformers, 2020.
Ba, L. J., Kiros, J. R., and Hinton, G. E. Layer normalization.
CoRR , 2016.
Bello, I., Zoph, B., Le, Q., Vaswani, A., and Shlens, J.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


CoRR , 2016.
Bello, I., Zoph, B., Le, Q., Vaswani, A., and Shlens, J.
Attention augmented convolutional networks. In 2019
IEEE/CVF International Conference on Computer Vision,
ICCV , 2019.
Bertasius, G. and Torresani, L. Classifying, segmenting, and
tracking object instances in video with mask propagation.
InThe IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) , June 2020.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M.,
Gray, S., Chess, B., Clark, J., Berner, C., McCandlish,
S., Radford, A., Sutskever, I., and Amodei, D. Language
models are few-shot learners. 2020.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


S., Radford, A., Sutskever, I., and Amodei, D. Language
models are few-shot learners. 2020.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov,
A., and Zagoruyko, S. End-to-end object detection with
transformers. In European Conference Computer Vision
(ECCV) , 2020.
Carreira, J. and Zisserman, A. Quo vadis, action recognition? A new model and the kinetics dataset. In 2017 IEEE
Conference on Computer Vision and Pattern Recognition,
CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , 2017.
Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., and
Zisserman, A. A short note about kinetics-600. CoRR ,
2018.
Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey,
W., Foster, G., Jones, L., Schuster, M., Shazeer, N., Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Chen,

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Z., Wu, Y ., and Hughes, M. The best of both worlds:
Combining recent advances in neural machine translation.
InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics . Association for
Computational Linguistics, 2018a.
Chen, Y ., Kalantidis, Y ., Li, J., Yan, S., and Feng, J. Aˆ2nets: Double attention networks. In Advances in Neural
Information Processing Systems 31 , 2018b.
Chen, Y ., Fan, H., Xu, B., Yan, Z., Kalantidis, Y ., Rohrbach,
M., Yan, S., and Feng, J. Drop an octave: Reducing
spatial redundancy in convolutional neural networks with
octave convolution. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision (ICCV) ,
October 2019.
Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. CoRR ,
2019.
Cordonnier, J., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In
8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,
2020 , 2020.
Dai, Z., Yang, Z., Yang, Y ., Carbonell, J., Le, Q., and
Salakhutdinov, R. Transformer-XL: Attentive language
models beyond a ﬁxed-length context. In Proceedings of
the 57th Annual Meeting of the Association for Computational Linguistics , 2019.
Deng, J., Dong, W., Socher, R., Li, L., Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database. In
2009 IEEE Conference on Computer Vision and Pattern
Recognition , pp. 248–255, 2009. doi: 10.1109/CVPR.
2009.5206848.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert:
Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 ,
2018.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT:

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


2018.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT:
Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer,
M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.
An image is worth 16x16 words: Transformers for image
recognition at scale. CoRR , 2020.
Fan, H., Li, Y ., Xiong, B., Lo, W.-Y ., and Feichtenhofer, C. Pyslowfast. https://github.com/
facebookresearch/slowfast , 2020.
Fan, Q., Chen, C.-F. R., Kuehne, H., Pistoia, M., and Cox,
D. More is less: Learning efﬁcient video representations
by big-little network and depthwise temporal aggregation.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


D. More is less: Learning efﬁcient video representations
by big-little network and depthwise temporal aggregation.
InAdvances in Neural Information Processing Systems ,
volume 32, 2019.
Feichtenhofer, C. X3d: Expanding architectures for efﬁcient
video recognition. CVPR , pp. 200–210, 2020.
Feichtenhofer, C., Fan, H., Malik, J., and He, K. Slowfast
networks for video recognition. In Proceedings of the
Is Space-Time Attention All You Need for Video Understanding?
IEEE/CVF International Conference on Computer Vision
(ICCV) , 2019a.
Feichtenhofer, C., Fan, H., Malik, J., and He, K. Slowfast
networks for video recognition. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV , 2019b.
Gavrilyuk, K., Sanford, R., Javan, M., and Snoek, C. G. M.
Actor-transformers for group activity recognition. In 2020
IEEE/CVF Conference on Computer Vision and Pattern
Recognition, CVPR , 2020.
Girdhar, R., Carreira, J., Doersch, C., and Zisserman, A.
Video action transformer network. In IEEE Conference on

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Girdhar, R., Carreira, J., Doersch, C., and Zisserman, A.
Video action transformer network. In IEEE Conference on
Computer Vision and Pattern Recognition, CVPR , 2019.
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P.,
Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y ., and
He, K. Accurate, large minibatch sgd: Training imagenet
in 1 hour. arXiv preprint arXiv:1706.02677 , 2017a.
Goyal, R., Kahou, S. E., Michalski, V ., Materzynska, J.,
Westphal, S., Kim, H., Haenel, V ., Fründ, I., Yianilos,
P., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I.,
and Memisevic, R. The "something something" video
database for learning and evaluating visual common sense.
CoRR , 2017b.
Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T.
Axial attention in multidimensional transformers. CoRR ,
2019.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Axial attention in multidimensional transformers. CoRR ,
2019.
Hu, H., Gu, J., Zhang, Z., Dai, J., and Wei, Y . Relation networks for object detection. In 2018 IEEE Conference on
Computer Vision and Pattern Recognition, CVPR , 2018.
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y ., and
Liu, W. Ccnet: Criss-cross attention for semantic segmentation. 2019.
Jiang, B., Wang, M., Gan, W., Wu, W., and Yan, J. Stm:
Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE/CVF International
Conference on Computer Vision (ICCV) , October 2019.
Kwon, H., Kim, M., Kwak, S., and Cho, M. Motionsqueeze:
Neural motion feature learning for video understanding.
InECCV , 2020.
Le, H., Sahoo, D., Chen, N., and Hoi, S. Multimodal transformer networks for end-to-end video-grounded dialogue
systems. In Proceedings of the 57th Annual Meeting of
the Association for Computational Linguistics , 2019.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


systems. In Proceedings of the 57th Annual Meeting of
the Association for Computational Linguistics , 2019.
Li, L., Chen, Y .-C., Cheng, Y ., Gan, Z., Yu, L., and
Liu, J. Hero: Hierarchical encoder for video+ language omni-representation pre-training. arXiv preprint
arXiv:2005.00200 , 2020a.Li, Y ., Li, Y ., and Vasconcelos, N. Resound: Towards action
recognition without representation bias. In The European Conference on Computer Vision (ECCV) , September
2018.
Li, Y ., Ji, B., Shi, X., Zhang, J., Kang, B., and Wang, L. Tea:
Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR) , June
2020b.
Lin, J., Gan, C., and Han, S. Tsm: Temporal shift module for
efﬁcient video understanding. In Proceedings of the IEEE
International Conference on Computer Vision , 2019.
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev,

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


International Conference on Computer Vision , 2019.
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev,
I., and Sivic, J. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video
Clips. In ICCV , 2019.
Ott, M., Edunov, S., Grangier, D., and Auli, M. Scaling
neural machine translation. In Proceedings of the Third
Conference on Machine Translation: Research Papers ,
2018.
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer,
N., Ku, A., and Tran, D. Image transformer. In Dy, J. G.
and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML , 2018.
Qiu, Z., Yao, T., Ngo, C.-W., Tian, X., and Mei, T. Learning spatio-temporal representation with local and global
diffusion. In CVPR , 2019.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever,
I. Improving language understanding by generative pretraining. 2018.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Radford, A., Narasimhan, K., Salimans, T., and Sutskever,
I. Improving language understanding by generative pretraining. 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and
Sutskever, I. Language models are unsupervised multitask
learners. 2019.
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in
vision models. In Advances in Neural Information Processing Systems , pp. 68–80, 2019.
Sevilla-Lara, L., Zha, S., Yan, Z., Goswami, V ., Feiszli,
M., and Torresani, L. Only time can tell: Discovering
temporal data for temporal modeling. In Proceedings
of the IEEE/CVF Winter Conference on Applications of
Computer Vision (WACV) , pp. 535–544, January 2021.
Simonyan, K. and Zisserman, A. Very deep convolutional
networks for large-scale image recognition. In ICLR ,
2015.
Is Space-Time Attention All You Need for Video Understanding?

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


networks for large-scale image recognition. In ICLR ,
2015.
Is Space-Time Attention All You Need for Video Understanding?
Stroud, J., Ross, D., Sun, C., Deng, J., and Sukthankar, R.
D3d: Distilled 3d networks for video action recognition.
InProceedings of the IEEE/CVF Winter Conference on
Applications of Computer Vision (WACV) , March 2020.
Sun, C., Myers, A., V ondrick, C., Murphy, K., and Schmid,
C. Videobert: A joint model for video and language
representation learning, 2019.
Szegedy, C., Liu, W., Jia, Y ., Sermanet, P., Reed, S.,
Anguelov, D., Erhan, D., Vanhoucke, V ., and Rabinovich,
A. Going deeper with convolutions. In Computer Vision
and Pattern Recognition (CVPR) , 2015.
Teed, Z. and Deng, J. RAFT: recurrent all-pairs ﬁeld transforms for optical ﬂow. In Computer Vision - ECCV 2020
- 16th European Conference, Glasgow, UK, August 23-28,
2020, Proceedings, Part II , 2020.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


- 16th European Conference, Glasgow, UK, August 23-28,
2020, Proceedings, Part II , 2020.
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y ., and
Paluri, M. A closer look at spatiotemporal convolutions
for action recognition. In 2018 IEEE Conference on
Computer Vision and Pattern Recognition, Salt Lake City,
USA, 2018 , 2018.
Tran, D., Wang, H., Feiszli, M., and Torresani, L. Video
classiﬁcation with channel-separated convolutional networks. ICCV , pp. 5551–5560, 2019.
van der Maaten, L. and Hinton, G. Visualizing data using t-SNE. Journal of Machine Learning Research , 9:
2579–2605, 2008. URL http://www.jmlr.org/
papers/v9/vandermaaten08a.html .
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information
Processing Systems , 2017a.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Processing Systems , 2017a.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information
Processing Systems 30 . 2017b.
Wang, H., Tran, D., Torresani, L., and Feiszli, M. Video
modeling with correlation networks. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020a.
Wang, H., Zhu, Y ., Green, B., Adam, H., Yuille, A. L., and
Chen, L. Axial-deeplab: Stand-alone axial-attention for
panoptic segmentation. In Computer Vision - ECCV 2020
- 16th European Conference , 2020b.
Wang, X., Girshick, R., Gupta, A., and He, K. Non-local
neural networks. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR) ,
June 2018a.Wang, X., Girshick, R. B., Gupta, A., and He, K. Non-local

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


June 2018a.Wang, X., Girshick, R. B., Gupta, A., and He, K. Non-local
neural networks. In 2018 IEEE Conference on Computer
Vision and Pattern Recognition, CVPR 2018, Salt Lake
City, UT, USA, June 18-22, 2018 , 2018b.
Wang, X., Xiong, X., Neumann, M., Piergiovanni, A. J.,
Ryoo, M. S., Angelova, A., Kitani, K. M., and Hua, W.
Attentionnas: Spatiotemporal attention cell search for
video classiﬁcation. In Computer Vision - ECCV 2020 16th European Conference, Glasgow, UK, August 23-28,
2020, Proceedings, Part VIII , 2020c.
Weissenborn, D., Täckström, O., and Uszkoreit, J. Scaling autoregressive video models. In 8th International
Conference on Learning Representations, ICLR , 2020.
Wightman, R. Pytorch image models. https://github.
com/rwightman/pytorch-image-models ,
2019.
Xie, S., Sun, C., Huang, J., Tu, Z., and Murphy, K.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


com/rwightman/pytorch-image-models ,
2019.
Xie, S., Sun, C., Huang, J., Tu, Z., and Murphy, K.
Rethinking spatiotemporal feature learning: Speedaccuracy trade-offs in video classiﬁcation. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV , pp. 318–335, 2018. doi: 10.1007/
978-3-030-01267-0\_19. URL https://doi.org/
10.1007/978-3-030-01267-0_19 .
Yang, Z., Garcia, N., Chu, C., Otani, M., Nakashima, Y ., and
Takemura, H. Bert representations for video question answering. In The IEEE Winter Conference on Applications
of Computer Vision , 2020.
Zhao, H., Jia, J., and Koltun, V . Exploring self-attention
for image recognition. In 2020 IEEE/CVF Conference on
Computer Vision and Pattern Recognition, CVPR , 2020.
Zhou, L., Zhou, Y ., Corso, J. J., Socher, R., and Xiong, C.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Zhou, L., Zhou, Y ., Corso, J. J., Socher, R., and Xiong, C.
End-to-end dense video captioning with masked transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018.

Accepted to ICML 2021

Layer Normalization

Language Models are Few-Shot Learners


Uniﬁed Graph Structured Models for Video Understanding
Anurag Arnab Chen Sun Cordelia Schmid
Google Research
faarnab, chensun, cordelias g@google.com
Abstract
Accurate video understanding involves reasoning about
the relationships between actors, objects and their environment, often over long temporal intervals. In this paper,
we propose a message passing graph neural network that
explicitly models these spatio-temporal relations and can
use explicit representations of objects, when supervision is
available, and implicit representations otherwise. Our formulation generalises previous structured models for video
understanding, and allows us to study how different design choices in graph structure and representation affect
the model’s performance. We demonstrate our method on
two different tasks requiring relational reasoning in videos
– spatio-temporal action detection on AVA and UCF10124, and video scene graph classiﬁcation on the recent Action Genome dataset – and achieve state-of-the-art results
on all three datasets. Furthermore, we show quantitatively
and qualitatively how our method is able to more effectively
model relationships between relevant entities in the scene.
1. Introduction
Deep learning has enabled rapid advances in many image understanding tasks such as image classiﬁcation [20],
object detection [46] and semantic segmentation [7]. However, progress on recent video understanding datasets such

Unified Graph Structured Models for Video Understanding

Summary:
Accurate video understanding involves reasoning about the relationships
between actors, objects and their environment, often over long temporal
intervals. In this paper, we propose a message passing graph neural network
that explicitly models these spatio-temporal relations and can use explicit
representations of objects, when supervision is available, and implicit
representations otherwise. Our formulation generalises previous structured
models for video understanding, and allows us to study how different design
choices in graph structure and representation affect the model's performance.
We demonstrate our method on two different tasks requiring relational reasoning
in videos -- spatio-temporal action detection on AVA and UCF101-24, and video
scene graph classification on the recent Action Genome dataset -- and achieve
state-of-the-art results on all three datasets. Furthermore, we show
quantitatively and qualitatively how our method is able to more effectively
model relationships between relevant entities in the scene.

1. Uncertainty-aware Weakly Supervised Action Detection from Untrimmed Videos

Anurag Arnab, Chen Sun, Arsha Nagrani, Cordelia Schmid

2. Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton

3. Object Level Visual Reasoning in Videos

Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, Greg Mori

4. Interaction Networks for Learning About Objects, Relations and Physics

Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende

5. Relational Inductive Biases, Deep Learning, and Graph Networks


object detection [46] and semantic segmentation [7]. However, progress on recent video understanding datasets such
as A V A [17] and Charades [49] has lagged behind in comparison. Further progress in the video understanding tasks
posed by these datasets would facilitate applications in autonomous vehicles, health monitoring and automated media
analysis and production among others.
A reason why video understanding is so challenging is
because, as shown in Fig. 1, it requires understanding the
interactions between actors, objects and other context in the
scene. Furthermore, these interactions are not always observable from a single frame, and thus require reasoning
over long temporal intervals. This is illustrated in Fig. 1,
where understanding the actions of the person in the centreframe is not possible from the target keyframe alone. In order to know that the woman is “listening”, we need to con-sider the man who is speaking but no longer in the scene.
And to correctly infer that the woman is “driving” the car,
rather than “riding” like the man, we must take into account
that she is later holding the steering wheel.
Video is a signiﬁcantly higher-dimensional signal than
single images, due to its additional temporal axis, and so we
believe learning these unlabelled interactions directly from
current datasets with large convolutional networks is not

2. Layer Normalization


single images, due to its additional temporal axis, and so we
believe learning these unlabelled interactions directly from
current datasets with large convolutional networks is not
feasible. In this paper, we propose a structured graph neural
network to explicitly model these spatio-temporal interactions. We model actors and objects (explicitly with bounding boxes when we have supervision, and implicitly otherwise) as nodes in our spatio-temporal graph and perform
message passing inference to directly model their relations.
Although a wide range of graph-structured models have
been proposed for action recognition, we note that there has
been no unifying formulation for these models. As such,
some works only model spatial relations between actors
and objects [14, 54], but not how these interactions evolve
over time. Other approaches model long-range temporal
interactions [61], but do not capture spatial relations and
are not trained end-to-end. And whilst some methods do
model spatio-temporal interactions of objects [3, 60], their
explicit representations of objects need additional supervision, and are not evaluated on spatio-temporal localisation
tasks which requires detailed understanding and is necessary for analysing untrimmed videos.
Our graph network formulation based on the messagepassing neural network [13] abstraction, allows us to explicitly model interactions between actors, objects and

2. Layer Normalization


Our graph network formulation based on the messagepassing neural network [13] abstraction, allows us to explicitly model interactions between actors, objects and
the scene, and how these interactions evolve over time.
Our ﬂexible model allows us to use explicit object representations from a pretrained Region Proposal Network
(RPN) [46], and/or implicitly from convolutional feature
maps without additional supervision. Moreover, our general
formulation allows us to interpret previous work [14, 54, 60,
61, 71] as special cases, and thus understand how different
design choices in object representation, graph connectivity
and message passing functions affect the model’s performance. We demonstrate our versatile model on two different tasks: spatio-temporal action detection on A V A [17]arXiv:2103.15662v1  [cs.CV]  29 Mar 2021
TimeTarget keyframeLabels: listen, drive, sit…………Figure 1: Understanding videos requires reasoning over long-term spatio-temporal interactions between actors, objects and the environment. The actions of the woman in the centre frame are ambiguous given the nearby frames which typical 3D CNN architectures consider.
However, by considering her interactions with the man from nearby frames, we know she is “listening” to him. And the fact that she is later

2. Layer Normalization


However, by considering her interactions with the man from nearby frames, we know she is “listening” to him. And the fact that she is later
holding the steering wheel indicates she is “driving” in contrast to the man who is “riding”. In this paper, we propose a spatio-temporal
graph on which we perform message passing to explicitly model these spatio-temporal interactions. Example from the A V A dataset [17].
and UCF101-24 [53], and video scene graph prediction on
the recent Action Genome [23] dataset. Both of these tasks
require modelling the spatio-temporal interactions between
the actors and/or objects in the scene, and we show consistent improvements from using each component of our
model and achieve state-of-the-art results on each dataset.
Furthermore, we observe that the largest improvements
in A V A are indeed achieved on action classes involving
human-to-human and human-to-object interactions, and visualisations of our network show that it is focusing on scene
context that is intuitively relevant to its action classiﬁcation.
2. Related Work
Modelling contextual relationships has a long history in
scene understanding. Relevant examples of early works in
this area included modelling interactions between humans
and objects [18, 68], different objects [45] and relationships

2. Layer Normalization


scene understanding. Relevant examples of early works in
this area included modelling interactions between humans
and objects [18, 68], different objects [45] and relationships
between human actions and scene context [41, 16]. Furthermore, it has also been shown that human vision is reliant on
context too [43]. In this paper, we consider video understanding tasks, speciﬁcally spatio-temporal action recognition and video scene graph parsing, which involve reasoning
about interactions between actors, objects and their environment in both space and time.
Early work in action recognition used hand-crafted features to encode motion information [32, 58]. Advances
in deep learning ﬁrst saw repurposing of 2D image convolutional neural networks (CNNs) for video as “two
stream” networks [27, 50] followed by spatio-temporal 3D
CNNs [6, 12, 55, 72]. However, these architectures focus on extracting coarse, video-level features and are not
suitable for learning the ﬁne-grained relations depicted in
Fig. 1. Consequently, whilst initial approaches to spatiotemporal action detection involved extending 2D object detectors [38, 46] temporally [26, 51, 1, 65], current leading
methods [14, 61, 71] on the A V A dataset [17] all explicitly

2. Layer Normalization


methods [14, 61, 71] on the A V A dataset [17] all explicitly
model relationships, with approaches which we show can
be interpreted as variants of graph neural networks.
Graph neural networks (GNNs) explicitly model relations between entities by modelling them as nodes in adirected or undirected graph [5, 48, 29], which interact
via a neighbourhood deﬁned for each node. The selfattention [56] and Non-local [59] operators can also be
thought of as GNNs [5, 13] where each element in a feature map is a node, and all nodes are fully-connected to
each other. Such attention-based models have excelled at
a number of natural language processing and computer vision tasks and have inspired many follow-up methods [21,
8, 9, 62, 70].
Many structured models have also recently been employed in video understanding. However, there has previously been no coherent framework to unify these approaches. Consequently, some works only model spatial relations between actors and objects, but not how these evolve
over time [54, 14]. And whilst LFB [61] models long range
temporal interactions, it does not capture spatial relationships within a keyframe. Moreover, in order to model longrange interactions, [61] uses precomputed features and is
thus not trained end-to-end. Our proposed method, which is

2. Layer Normalization


thus not trained end-to-end. Our proposed method, which is
based on the message passing neural network (MPNN) [13]
framework, coherently models both spatial and temporal
interactions. After describing our model in Sec. 3, we
show how previous structured models for action detection [54, 14, 60, 61, 71] can be thought of as special cases
of our model. Furthermore, our ﬂexible model can reason
about objects both when we have explicit supervision for
them, and when we do not. Prior work in video understanding, which have proposed less generic models, have either
assumed the case of explicit object supervision [3, 60] or
not [54, 14, 71], but have not considered the scenario when
both options are available. Furthermore, our general formulation allows us to ablate graph modelling design choices,
such as the object representation, message passing functions and temporal context, in a manner not possible with
these previous, more speciﬁc approaches. Moreover, we
note that some approaches which have employed spatiotemporal graphs in video have only considered frame-toframe interactions [22, 60, 3, 40, 42, 44], rather than longrange relations, and have not evaluated on spatio-temporal
localisation which requires more detailed understanding
3D CNN
sit, answer phone, carry/hold
object

2. Layer Normalization


localisation which requires more detailed understanding
3D CNN
sit, answer phone, carry/hold
object
sit, touch (an object), watch (a
person), talk (a person)
sit, touch (an object), watch (a
person), listen (a person)
Readout Function,

Spatio
temporal graph
construction and message passing
Foreground node
Implicit context
Explicit contextFigure 2: Overview of our method: We construct a spatio-temporal graph, and perform message-passing inference on it, to model interactions between actors, objects and their environment. Foreground nodes (circles) have readout functions associated with them for the task of
interest ( i.e. for action recognition, the nodes represent person bounding boxes which are classiﬁed into actions). Context nodes (squares)
model additional information, and can either be implicit, as cells of the original feature map, or explicit by ROI-pooling external region
proposals (RPN [46] not shown for clarity). The initial state of each node is a spatio-temporal feature vector extracted from a 3D CNN.
and is essential for analysing untrimmed videos.
We also note that scene graph parsing [25, 30] is another task that evaluates a model’s ability to reason about

2. Layer Normalization


and is essential for analysing untrimmed videos.
We also note that scene graph parsing [25, 30] is another task that evaluates a model’s ability to reason about
the interactions between different objects by representing
objects as nodes and relationships as edges in a graph. Although the task was originally posed for single images [30],
the recent Action Genome [23] dataset extends this task
to video by adding annotations to Charades [49]. While
GNN-based approaches have also been used in scene graph
parsing [10, 33, 64, 66] for single images, to our knowledge, they have not been applied to model temporal relations in video. Moreover, the same model has not also been
demonstrated on spatio-temporal action recognition like our
method.
3. Proposed Approach
Our model aims to build a structured representation of
a video by representing it as a graph of actors, objects
and contextual elements in the scene, as shown in Fig. 2.
This structured representation is then used to perform tasks
which require understanding the interactions between the
elements in the graph, such as action recognition and scene
graph prediction. Note that we do not assume we have annotations for relevant scene context.
Our approach is based on Message Passing Neural Networks (MPNN) [13], as it is a ﬂexible framework that generalises many previous graph neural network algorithms [4,

2. Layer Normalization


Our approach is based on Message Passing Neural Networks (MPNN) [13], as it is a ﬂexible framework that generalises many previous graph neural network algorithms [4,
5, 29, 57] We review this approach, and describe how we
adapt it for video understanding in Sec. 3.1, before detailing our model in Sec. 3.2 through 3.5. Finally, we dis-cuss how previous structured models for video understanding [14, 54, 60, 61, 71] can be regarded as speciﬁc instantiations of our model in Sec. 3.6.
3.1. Message Passing Neural Networks (MPNNs)
MPNNs operate on a directed or undirected graph, G
consisting of nodes, v2V, and a neighbourhood for each
node,Nv, that deﬁnes the graph’s connectivity. For video
models we distinguish between the spatial, Sv, and temporal,Tv, neighbourhoods for node v(Nv=Sv[Tv). Each
node,v, is associated with a latent state, hv. Inference in
this model consists of a message passing phase, and a ﬁnal readout phase. In the message passing phase, messages
for each node, mv, are ﬁrst computed by applying spatial

2. Layer Normalization


for each node, mv, are ﬁrst computed by applying spatial
and temporal message passing functions, MsandMtrespectively, to all nodes in its neighbourhood as described in
Eq. (1). An update function, U, then aggregates the received
messages to update the latent state, hv,
mi+1
v=X
w2SvMs(hi
v;hi
w;i
s) +X
u2TvMt(hi
v;hi
u;i
t)(1)
hi+1
v=U(mi+1
v;hi
v); (2)
wheredenotes learnable function parameters. Intuitively,
the state of a node, hv, is updated by aggregating the messages passed from its neighbours. Finally, after I1iterations of message passing, a readout function, R, uses the
updated node features for the classiﬁcation task of interest
y=R(fhi
vgjv2G): (3)
As illustrated in Fig. 2, our graph consists of a set of
“Foreground” nodes, F=ff1;f2;:::;fngand “Context”

2. Layer Normalization


“Foreground” nodes, F=ff1;f2;:::;fngand “Context”
nodes,C=fc1;c2;:::;cmgwherenandmvary for each
video. The “Foreground” nodes, F, have readout and loss
functions associated with them, and correspond to the objects that will subsequently be classiﬁed. For the task of action recognition,Fcorresponds to bounding boxes of each
actor in the keyframe. Whilst for scene graph prediction, F
refers to bounding boxes for all potential objects of interest. The “Context” nodes, C, capture additional information
extracted from the scene for relational reasoning. The representations of these nodes, and their spatial connectivity,
are detailed next.
3.2. Spatial Model
The spatial connections in our graph model relationships between actors, objects and scene context in the same
frame. For example, recognising an action such as “give
object to a person” in the A V A dataset [17] requires understanding both people involved in the action and also the object being transferred.
To model such interactions, we ﬁrst represent the foreground nodes of our model, F, by extracting convolutional features from the last layer of the network, X2

2. Layer Normalization


To model such interactions, we ﬁrst represent the foreground nodes of our model, F, by extracting convolutional features from the last layer of the network, X2
Rthwc, and using ROI-Align [19] followed by spatiotemporal pooling and a linear projection to obtain fi2Rd.
We model scene context by considering the features from
each spatial position in the feature map, X, as a contextual
node,ci, in our graph. Note that these features are projected
toRd. A similar representation was used by [54], which we
refer to as an implicit object model as it enables the network
to encode information about the scene and relevant objects
without any extra supervision. This approach is also known
as “grid features” in visual question answering [4, 24].
It is also possible to augment our set of contextual nodes
with an explicit object representation by computing classagnostic object proposals with a Region Proposal Network
(RPN) [46]. We use an RPN pretrained on the OpenImages
dataset [31] and obtain a d-dimensional feature from each
proposal by using ROI-Align and a linear projection as for
the foreground nodes. A similar idea of using explicit object
representations in video has also been employed by [3] and

2. Layer Normalization


the foreground nodes. A similar idea of using explicit object
representations in video has also been employed by [3] and
[60], though not for spatio-temporal action recognition.
Messages are then computed and passed to foreground
nodes from both foreground and contextual nodes in the
graph. We only update the foreground nodes during message passing, as these are the nodes which are subsequently
classiﬁed in the ﬁnal Readout phase. Concretely, the spatial
neighbourhood for each foreground node in our graph is
Sv=F[C8v2F: (4)
ContextForegroundFrames
⚫
⚫
⚫𝜏"𝜏#Figure 3: Illustration of temporal connectivity. Foreground (blue)
and context (green) nodes have initial states computed from spatiotemporal features over tframes (shaded in yellow), centred on
a keyframe (denoted by a black circle). scontrols the distance
between selected keyframes; cis the total number of keyframes
considered. Here, c= 3,s= 7frames andt= 5 frames.
3.3. Temporal Model
We also include temporal connections in our graph to
model long-range interactions between actors and objects.

2. Layer Normalization


3.3. Temporal Model
We also include temporal connections in our graph to
model long-range interactions between actors and objects.
As shown in Fig. 1, understanding actions often requires
reasoning about actors who are no longer visible in the current frame, thus requiring large temporal contexts.
We model these temporal interactions by connecting
foreground nodes in keyframe twith all other foreground
nodes in neighbouring keyframes t02T . Concretely, we
deﬁne the temporal context, c, and temporal stride, s, hyperparameters. As shown in Fig. 3, cis the total number of
number of keyframes in the video which we consider in our
temporal graph. sis the sampling rate at which we select
keyframes, as s1allows us to consider a wider temporal
interval in a more computationally efﬁcient manner. This is
necessary to train the entire model end-to-end. Moreover,
as each foreground feature node in the graph, fi, is itself
a spatio-temporal feature computed over a period of tby
a 3D CNN, selecting adjacent keyframes (and effectively
settings= 1 keyframe) could result in redundant information being captured by the temporal connections in the

2. Layer Normalization


a 3D CNN, selecting adjacent keyframes (and effectively
settings= 1 keyframe) could result in redundant information being captured by the temporal connections in the
graph. Note that our deﬁnition of keyframe follows common datasets [17, 23] as a frame we aim to classify given
surrounding temporal context.
More formally, we can describe the neighbourhood of
each foreground node, v, for temporal message passing as
Tv=bc=2c[
dt=c=2eFts: (5)
Here, we use the superscript to denote the temporal index,
and without loss of generality consider t= 0 to be the
centre keyframe, meaning that negative time indices correspond to frames in the past. We set cto be an odd, positive
integer to employ an equal-sized temporal window on either
side of the centre keyframe.
We ﬁrst perform spatial message passing, before passing messages temporally. This allows information from the
context nodes in frame t0,ct0
j, to efﬁciently propagate to a
foreground node ft
i(wheret6=t0) viaft0
j, as the foreground
nodes are fully-connected temporally.
The following section now describes the messages that
are passed along the graph described above.

2. Layer Normalization


j, as the foreground
nodes are fully-connected temporally.
The following section now describes the messages that
are passed along the graph described above.
3.4. Message Passing Functions
We ﬁrst observe that the Non-local operator or selfattention [56, 59] can be considered as a message passing
function in a fully-connected graph, where each node, hv,
is an element in the input feature map, H, and the neighbourhood comprises of all other feature elements. And
since self-attention is employed with a residual connection [59, 56] and layer normalisation [2, 56], the update
function of Non-local when viewed as an MPNN is
U=LN(hv+Self-Attention (H)): (6)
Similar analysis [5, 13] has shown that Graph Attention
Networks (GAT) [57], Relational Networks [47] and many
other graph neural networks [48, 29, 35, 4] can also be interpreted as MPNNs. In this paper, we consider Non-local [59]
and Graph Attention (GAT) [57] as message passing functions within the update function, Eq. (6).
Non-local We modify Non-local [59] to pass messages
from all Foreground and Context nodes to only Foreground

2. Layer Normalization


Non-local We modify Non-local [59] to pass messages
from all Foreground and Context nodes to only Foreground
nodes (Eq. 4), as these are the nodes which are used for the
ﬁnal classiﬁcation in the Readout phase,
M=SoftmaxQK>
p
d
V (7)
Q=AWqK= [AjjC]WkV= [AjjC]Wv:
Here,A2RndandC2Rmdare matrices where each
row is a foreground and context feature node respectively,
Mis theRndmatrix of messages received by each node in
A,[AjjC]denotes the concatenation of these matrices and
Wq,WkandWvare learnable ddprojection matrices.
Graph attention (GAT) The graph attention [57] message for a node vis computed as
mv=0
@X
j2NvijWahj1
A (8)
ij=Softmax

w>
b[hijjhj]
(9)
whereis a ReLU non-linearity and Waandwbare a
learnable matrix and vector respectively.

2. Layer Normalization


b[hijjhj]
(9)
whereis a ReLU non-linearity and Waandwbare a
learnable matrix and vector respectively.
Parallel messages It is also possible to compute multiple
incoming messages in parallel for a node, hv. When usingmultiple Non-local or Graph Attention functions, this corresponds to multi-headed attention [56, 57]. By viewing Nonlocal and GAT as message passing functions, we can also
aggregate messages from a combination of these two. In
these cases, we aggregate the messages using an attentionweighted convex combination, as performed in Eq. (9).
3.5. Readout Function
AfterI1iterations of message passing, a readout
function is applied on the Foreground nodes to obtain the
ﬁnal predictions. For action detection, the readout function is a linear classiﬁer operating on each element of F,
where each foreground node, fi, corresponds to the features
of an actor in the keyframe. For scene graph prediction, the
readout function consists of two classiﬁers: The ﬁrst linear
classiﬁer predicts the object-class label of each foreground
node. The second is a function of each pair of foreground
nodes, and predicts the relationship label between them.
3.6. Discussion

2. Layer Normalization


node. The second is a function of each pair of foreground
nodes, and predicts the relationship label between them.
3.6. Discussion
We note that many previous structured models for video
understanding can be considered as special cases of our proposed MPNN framework:
Girdhar et al . [14] only consider a spatial model, i.e.
N=S, using implicit objects and Non-local [60] for
message passing. ACRN [54] has the same graph structure, but uses Relational Networks [47] for message passing instead. LFB [61], in contrast, considers only a temporal model, i.e.N=T, using Non-local as the message passing function in a graph that is fully-connected in
time. However, [61] do not consider a spatial model to capture interactions between actors in the keyframe. Zhang et
al. [71], on the other hand, model both temporal and spatial connections. However, they effectively model three
separate graphs: the ﬁrst models actors in short, 3-second
tubelets using GCN [29] for message passing. The other
two graphs model actor-actor and actor-object relations using a message passing method similar to GAT [57], but using a hand-deﬁned weighting function rather than a learned

2. Layer Normalization


two graphs model actor-actor and actor-object relations using a message passing method similar to GAT [57], but using a hand-deﬁned weighting function rather than a learned
one as in GAT (Eq. 9). Wang et al. [60] also model a spatiotemporal graph, using GCN for message passing, and an explicit object representation in the spatial model. However,
their temporal connections are only among adjacent frames,
which does not allow information propagation between all
frames in long sequences, in contrast to our model which
is fully-connected temporally. With our uniﬁed framework,
we study the effect of graph modelling design choices and
show how we outperform previous work next in Sec. 4.
4. Experiments
We evaluate our method’s ability to model spatiotemporal interactions with experiments on spatio-temporal
action detection and video scene graph classiﬁcation.
Table 1: Ablation study on Action Genome using a 3D ResNet 50 backbone. We report the effect of (a) different message passing functions,
(b) temporal connections in the graph ( c= 1;s= 1corresponds to only spatial message passing) and (c) iterations of message passing.
(a) Spatial message passing functions
SGCls PredCls

2. Layer Normalization


2. Layer Normalization


Iterations R@20
1 51.1
2 51.6
3 51.6
5 51.8
Spatio-temporal action recognition We evaluate on
A V A [17], the largest dataset for this task consisting of
15-minute video clips obtained from movies, and UCF10124 [53], the previous standard benchmark for this task. A V A
is labelled with atomic actions, where one person is typically performing multiple actions simultaneously, whilst actors perform only a single high-level action in UCF101-24.
We follow standard protocol and evaluate using the Frame
AP at an IoU threshold of 0.5 on both datasets. For A V A, we
use v2.2 annotations for ablations, and either v2.1 or v2.2
annotations for fair comparisons with prior work.
Video scene graph prediction We evaluate on the recent Action Genome dataset [23], which adds scene graph
annotations to Charades [49], on two scene graph tasks:
scene graph classiﬁcation (SGCls) and predicate classiﬁcation (PredCls). Both tasks are evaluated using the standard
Recall@K metric [39, 23] (R@K) which measures the fraction of ground truth relationship triplets (subject-predicateobject) that appear in the top K scoring predicted triplets. In

2. Layer Normalization


SGCls, ground truth bounding box co-ordinates are given,
and the aim is to predict their object classes, as well as relationship labels between pairs of objects. PredCls is simpler,
as both bounding box co-ordinates and object classes are
given, and only the relationship label must be predicted.
Implementation details We use the public implementation of SlowFast [11] as our baseline with the 3D ResNet
50 or ResNet 101 backbones [20], as it is the current stateof-the-art. The network is similar to Fast-RCNN [15] as it
uses external region proposals to extract features from the
last feature map of res 5using ROI-Align [19]. These features are then spatio-temporally pooled and classiﬁed.
In our graph model, we initialise the internal states, hi,
of our Foreground nodes using these ROI-Aligned res 5features. For our action detection experiments, the Foreground
nodes in our graph correspond to bounding boxes of actors,
and we use the same person detector as [11, 61] for our actor region proposals. For scene graph experiments, we use
ground truth boxes of people and objects as our Foreground
nodes as we evaluate scene graph classiﬁcation (SGCls) and

2. Layer Normalization


ground truth boxes of people and objects as our Foreground
nodes as we evaluate scene graph classiﬁcation (SGCls) and
predicate classiﬁcation (PredCls).Unless otherwise speciﬁed, we use SlowFast 88which
corresponds to 32 input frames where the video is subsampled by a factor of 2. This means that each feature node,
hi, in the graph aggregates t= 2:1seconds of temporal
information on A V A as the videos are sampled at 30 frames
per second (fps). As keyframes in A V A are deﬁned at 1 fps,
it means that we need to set s2keyframes for the temporal information captured by temporally-adjacent nodes in
the graph to not overlap. For Action Genome, the Charades
videos are sampled at 24 fps and the keyframes are on average 0.85 seconds, or 20.5 frames apart.
We train our network for 20 epochs using synchronous
SGD on 8 GPUs and a total batch size of 64, initialising from a Kinetics-400 [28] pretrained model. The baseline model in all our experiments is SlowFast without any
graph module trained in an identical manner. When training
spatio-temporal graph models, we ﬁnetune for 10 epochs

2. Layer Normalization


graph module trained in an identical manner. When training
spatio-temporal graph models, we ﬁnetune for 10 epochs
from a model trained with only a spatial graph and reduce
the batch size by a factor of c. We also “freeze” batch
normalisation statistics as they have high variance for small
batches [19]. Full training details are in the appendix.
4.1. Experiments on video scene graph prediction
Spatial message passing We ﬁrst ablate only the spatial component (Sec. 3.2) of our model in Tab. 1a. Graph
Attention (GAT) [57] performs slightly better than Nonlocal [56, 59] as a message passing function, and we obtain
a further small improvement by combining the two methods
in parallel (Sec. 3.4). Overall, we improve upon our baseline model, which is SlowFast based on ResNet 50 without
any graph modelling, by 2.4 points for R@20 on SGCls.
Another baseline is to insert a Non-Local layer [59] after the ﬁnal res 5layer and before the ROI-Align layer, since
Non-local can also be viewed as a graph network (Sec. 3.4).

2. Layer Normalization


Non-local can also be viewed as a graph network (Sec. 3.4).
In this case, the Foreground and Context nodes are not explicitly modelled as in our method and the overall performance is 2.2 points less than our method for the R@20 of
SGCls. This suggests that explicit modelling of the Foreground nodes which are subsequently classiﬁed is important to performance. We note, however, that Non-Local [59]
is typically employed earlier in a network ( i.e. in res 3or
res4[59, 61, 11]) and can thus be seen as a complementary
Table 2: Comparison to existing methods on the Action Genome
dataset [23]. Previous methods reported by [23].
SGCls PredCls
R@20 R@50 R@10 R@20
MSDN [34] 44.0 47.2 – –
IMP [64] 44.1 47.4 – –
RelDN [69] 46.7 49.4 – –
SlowFast (ResNet 50) 48.9 51.3 78.7 93.8
Ours (ResNet 50) 53.8 56.0 79.3 94.2
personsofacouch
table
laptopnot looking atnot contacting
not contactingnot looking at
not contacting
in front of
looking aton the side of

2. Layer Normalization


personsofacouch
table
laptopnot looking atnot contacting
not contactingnot looking at
not contacting
in front of
looking aton the side of
in front of
person
sofacouchtelevision
laptop
table
not looking atnot contactingin front of
not contacting
behindlooking at
not looking atbeneathlooking at
in front of
sitting on
on the side ofabove
touching
Figure 4: Example scene graph predictions for consecutive
keyframes of Action Genome [23]. Classiﬁed bounding boxes are
on top, and the corresponding scene graph (objects are shown as
coloured rectangles, and relations with light-blue ovals) below.
method to improve features learned by the network.
Temporal message passing Table 1b adds temporal connections (Sec. 3.3) to our model, using GAT for message
passing, as it outperformed Non-local in Tab. 1a. We observe consistent improvements for a wide range of temporal
contexts,c, and strides, s, showing the utility of modelling
temporal dynamics. We improve over a graph with only
spatial connections by as much as 2.7 points for R@20.
For a temporal model, another baseline to consider is
to simply increase the temporal information available to
the spatial-only graph model by increasing the number of

2. Layer Normalization


For a temporal model, another baseline to consider is
to simply increase the temporal information available to
the spatial-only graph model by increasing the number of
frames at the input. When tripling the number of input
frames, the R@20 is 52.4, less than all variants of our temporal model with c= 3. Similarly, increasing input frames
by a factor of 5, the R@20 is 52.8, less than all our temporal models with c= 5. This improvement demonstrates the
beneﬁts of message passing on an explicit temporal graph.
Iterations of message passing Table 1c shows that the
model’s performance plateaus after 2 iterations of message
passing, where parameters are not tied across iterations.
Comparison to state-of-the-art Finally, we compare to
prior works in Tab. 2 which are single-image models evalu-Table 3: Analysis of message passing neighbourhood on A V A using SlowFast with a ResNet-50 backbone as the baseline. By varying the neighbourhood, we study the effect of object representations (implicit, explicit or none) and temporal connections. We
report the Frame mAP on the three types of action classes in A V A.
Message passing neighbourhood PoseHumanHumanHumanObjectAll
SlowFast baseline (none) 43.1 25.2 17.4 24.8

2. Layer Normalization


Message passing neighbourhood PoseHumanHumanHumanObjectAll
SlowFast baseline (none) 43.1 25.2 17.4 24.8
Actors only 43.2 27.0 17.8 25.6
Implicit objects only 43.4 26.7 18.0 25.7
Explicit objects only 43.0 26.7 17.8 25.5
Actor + Implicit 43.4 26.8 18.3 25.9
Actor + Implicit + Explicit 43.7 27.0 18.4 26.1
Spatio-temporal 43.8 27.5 19.9 27.0
ated by [23]. Our SlowFast, ResNet 50 3D baseline outperforms these, showing the importance of using spatiotemporal features for this task. As Action Genome contains
videos of humans acting to scripts, there is temporal structure in the interactions between actors and objects in the
scene (Fig. 4). Our ﬁnal spatio-temporal graph structured
model improves substantially upon this baseline by 4.9 and
4.7 points for the R@20 and R@50 for SGCls respectively.
Our improvements over the baseline for PredCls are less, as
this task is easier and the performance is saturated.
4.2. Experiments on action recognition
Graph structure and object representation Table 3
compares the effect of changing the neighbourhood that

2. Layer Normalization


this task is easier and the performance is saturated.
4.2. Experiments on action recognition
Graph structure and object representation Table 3
compares the effect of changing the neighbourhood that
passes messages to each foreground node that is subsequently classiﬁed. We report the performance across the
three types of action categories in A V A [17] – Pose, HumanHuman and Human-Object – to show the effect that different object representations (implicit, explicit or neither as
described in Sec. 3.2) and also temporal connections have.
Our SlowFast baseline does not model any explicit graph
and hence effectively performs no message passing. For
our A V A experiments, Foreground nodes in the graph correspond to bounding boxes of the actors in the scene, using
the same person detections as [61, 11]. Thus, in this section,
“Foreground” and “actor” nodes are used interchangeably.
We observe from the second row of Tab. 3 that passing messages only between actor nodes (and thus not modelling object interactions) provides an overall improvement
of 0.8 points. The largest gain, as expected, are for HumanHuman action classes, as these are the interactions modelled by passing messages between actor nodes in the graph.
When passing messages from implicit context nodes to
the actors (third row), we observe an improvement in all

2. Layer Normalization


When passing messages from implicit context nodes to
the actors (third row), we observe an improvement in all
types of action classes, but primarily in Human-Human and
Human-Object classes. This is because implicit context
nodes encompass the entire feature map, and thus capture
information about the whole scene (as also shown in Fig. 5).
Modelling objects explicitly with external regional proTable 4: Comparison on A V A. We report the Mean AP using v2.1
and v2.2 annotations. All methods pretrained on Kinetics 400.
“Multiscale” refers to averaging results over three scales [61].
Method v2.1 v2.2
ACRN (S3D) [54] 17.4 –
Zhang et al. (R50) [71] 22.2 –
SlowFast baseline (R50) 24.5 24.8
Girdhar et al. (I3D) [14] 25.0 –
LFB (R50) [61] 25.8 –
Ours (R50) 26.5 27.0
Ours Multiscale (R50) 27.3 27.7
SlowFast baseline (R101) 26.3 26.7
LFB (R101) [61] 26.8 –
Ours (R101) 28.3 28.8

2. Layer Normalization


SlowFast baseline (R101) 26.3 26.7
LFB (R101) [61] 26.8 –
Ours (R101) 28.3 28.8
LFB Multiscale (R101) [61] 27.7 –
Ours Multiscale (R101) 29.5 30.0
posals (fourth row) improves on the same action types as the
implicit object model, but performs marginally worse. This
suggests that our RPN trained on OpenImages [31] is unable
to detect the objects that are most discriminative of A V A actions. As object annotations are not provided in A V A, it is
not possible to evaluate the recall of our external region proposals on relevant objects. It is not clear if we should expect
more improvement from explicit objects, because although
previous works have considered implicit [14, 54, 71] and
explicit representations [3, 60], we are not aware of any that
have compared the two. Note that these explicit context
nodes also model Human-Human interactions as our RPN
is trained on people in OpenImages [31]. We obtain further
improvements by combining messages from actors, implicit
and explicit context nodes, as shown by the next two rows.
Finally, we evaluate our spatio-temporal model, which
passes messages from actors, implicit- and explicit-context

2. Layer Normalization


and explicit context nodes, as shown by the next two rows.
Finally, we evaluate our spatio-temporal model, which
passes messages from actors, implicit- and explicit-context
nodes withc= 3ands= 3, corresponding to an overall
temporal window of 8.5 seconds. This model performs the
best, improving upon the baseline by 2.2 points, or a relative
improvement of 8.9%. In particular, the temporal connections help to improve on Human-Object action classes.
A V A is a long-tailed dataset, and the most common failure modes of all variants of our model are the classes with
few training examples. However, our model improves on
both head and tail classes with respect to the baseline. This,
and detailed per-class results, are in the appendix.
State-of-the-art-comparison on A V A Table 4 compares
to recent, published work on A V A. Our method builds upon
SlowFast [11] as the base architecture, and our graph model
shows substantial improvements with either a 3D ResNet 50
or ResNet 101 backbone. As discussed in Sec. 3.6, Girdharet al. [14] and LFB [61] can be considered as special

2. Layer Normalization


or ResNet 101 backbone. As discussed in Sec. 3.6, Girdharet al. [14] and LFB [61] can be considered as special
cases of our graph model as they model only spatial andTable 5: Comparison to state-of-the-art on UCF101-24. We report
the Frame AP at 0.5 using the corrected annotations of [51].
Method Modality Mean AP
ACT [26] RGB + Flow 69.5
Song et al. [52] RGB + Flow 72.1
STEP [67] RGB + Flow 75.0
Guet al. [17] RGB + Flow 76.3
MOC [36] RGB + Flow 78.0
SlowFast (R50) RGB 76.6
SlowFast (R101) RGB 77.4
Ours (R50) RGB 78.6
Ours (R101) RGB 79.3
temporal edges respectively. Our method, which constructs
a spatio-temporal graph, outperforms both. Note that we
outperform LFB [61] across both ResNet 50 and ResNet
101 backbones. Zhang et al . [71] also model a spatiotemporal interactions, but employ three separate graphs
with a hand-crafted aggregation function which we also outperform with our single, coherent spatio-temporal graph.

2. Layer Normalization


with a hand-crafted aggregation function which we also outperform with our single, coherent spatio-temporal graph.
State-of-the-art comparison on UCF101-24 Table 5
shows that we outperform recent, published work on
UCF101-24 [53], using either a 3D ResNet 50 or ResNet
101 backbone. We do so without using optical ﬂow as an
additional input modality, showing that our network can
capture temporal information without it. Moreover, the improvements obtained from our graph model are consistent
with our results on the Action Genome and A V A datasets.
Qualitative results We visualise the spatial and temporal messages received by an actor node in the graph, on the
A V A dataset, when using GAT as the message passing function. In particular, we visualise the attention weights, ijin
(9), on each implicit object node (Fig. 5) and actor in neighbouring keyframes (Fig. 6), for a given actor. We observe
that the network places more weight on graph nodes that are
intuitively consistent with the ﬁnal action prediction. For
example, in Fig. 5, the model focuses on the actor’s face
when his action is “talk to” and the other person’s face when
his action is “listen to”.

2. Layer Normalization


when his action is “talk to” and the other person’s face when
his action is “listen to”.
5. Conclusion and Future Work
We have presented a novel spatio-temporal graph neural network framework to explicitly model interactions between actors, objects and their environment. Our formulation can model objects either implicitly or explicitly, and
generalises previous structured models for video understanding [14, 54, 61, 71]. Using our versatile approach, we
have achieved state-of-the-art results on two diverse tasks
across three datasets. Future work remains to harness explicit object representations more effectively on A V A.
(a)talk to (0.6), sit (0.92), watch person (0.81) (b)listen to person (0.9), watch person (0.89), sit (0.97) (c)carry/hold (1.0), sit(0.88)
Figure 5: Visualisation of spatial message passing when using implicit objects as detailed in Sec. 4.2. Note in (a) and (b) that the network
focuses on the selected actor (denoted by the green box on the left) when the action is “talk to”, and on the person speaking to him when

2. Layer Normalization


focuses on the selected actor (denoted by the green box on the left) when the action is “talk to”, and on the person speaking to him when
the action is “listen to person”. Attention weights, ijare colour-coded according to the bar on the far right. Best viewed in colour.
talk to (0.85), watch a person (0.82), sit(0.51)
Figure 6: Visualisation of temporal graph as detailed in Sec. 4.2. We show the neighbours for the actor depicted by the red box in the
centre keyframe, colour-coding the respective boxes with their attention weights ( ij). Predicted action scores of the red actor are below,
and selected keyframes are s= 2:1seconds apart. Notice that the model focuses on the people 2 keyframes (4.2 seconds) on either side
of the actor to recognise that she is “talking to” and “watching” them.
References
[1] Anurag Arnab, Chen Sun, Arsha Nagrani, and Cordelia
Schmid. Uncertainty-aware weakly supervised action detection from untrimmed videos. In ECCV , 2020. 2
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.

2. Layer Normalization


[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization. In arXiv preprint arXiv:1607.06450 ,
2016. 5
[3] Fabien Baradel, Natalia Neverova, Christian Wolf, Julien
Mille, and Greg Mori. Object level visual reasoning in
videos. In ECCV , 2018. 1, 2, 4, 8
[4] Peter Battaglia, Razvan Pascanu, Matthew Lai, and
Danilo Jimenez Rezende. Interaction networks for learning
about objects, relations and physics. In NeurIPS , 2016. 3, 4,
5
[5] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases,
deep learning, and graph networks. In arXiv preprint
arXiv:1806.01261 , 2018. 2, 3, 5
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action
recognition? a new model and the kinetics dataset. In CVPR ,
2017. 2

2. Layer Normalization


recognition? a new model and the kinetics dataset. In CVPR ,
2017. 2
[7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian
Schroff, and Hartwig Adam. Encoder-decoder with atrous
separable convolution for semantic image segmentation. In
ECCV , 2018. 1
[8] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng
Yan, and Jiashi Feng. Aˆ 2-nets: Double attention networks.
InNeurIPS , 2018. 2
[9] Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan
Shuicheng, Jiashi Feng, and Yannis Kalantidis. Graph-based
global reasoning networks. In CVPR , 2019. 2
[10] Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In CVPR , 2017.
3[11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and
Kaiming He. Slowfast networks for video recognition. In
ICCV , 2019. 6, 7, 8, 12
[12] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes.

2. Layer Normalization


ICCV , 2019. 6, 7, 8, 12
[12] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes.
Spatiotemporal residual networks for video action recognition. In NeurIPS , 2016. 2
[13] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol
Vinyals, and George E Dahl. Neural message passing for
quantum chemistry. In ICML , 2017. 1, 2, 3, 5
[14] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In CVPR , 2019.
1, 2, 3, 5, 8
[15] Ross Girshick. Fast r-cnn. In ICCV , 2015. 6
[16] Georgia Gkioxari, Ross Girshick, and Jitendra Malik. Contextual action recognition with r* cnn. In ICCV , 2015. 2
[17] Chunhui Gu, Chen Sun, David A Ross, Carl V ondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan,
George Toderici, Susanna Ricco, Rahul Sukthankar, et al.
Ava: A video dataset of spatio-temporally localized atomic

2. Layer Normalization


George Toderici, Susanna Ricco, Rahul Sukthankar, et al.
Ava: A video dataset of spatio-temporally localized atomic
visual actions. In CVPR , 2018. 1, 2, 4, 6, 7, 8, 12
[18] Abhinav Gupta, Aniruddha Kembhavi, and Larry S Davis.
Observing human-object interactions: Using spatial and
functional compatibility for recognition. PAMI , 31(10),
2009. 2
[19] Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Girshick. Mask r-cnn. In ICCV , 2017. 4, 6
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In CVPR ,
2016. 1, 6
[21] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR , 2018. 2
[22] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh
Saxena. Structural-rnn: Deep learning on spatio-temporal
graphs. In CVPR , 2016. 2
[23] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos

2. Layer Normalization


graphs. In CVPR , 2016. 2
[23] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos
Niebles. Action genome: Actions as compositions of spatiotemporal scene graphs. In CVPR , 2020. 2, 3, 4, 6, 7, 12
[24] Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik LearnedMiller, and Xinlei Chen. In defense of grid features for visual
question answering. In CVPR , 2020. 4
[25] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li,
David Shamma, Michael Bernstein, and Li Fei-Fei. Image
retrieval using scene graphs. In CVPR , 2015. 3
[26] Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari,
and Cordelia Schmid. Action tubelet detector for spatiotemporal action localization. In ICCV , 2017. 2, 8
[27] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas
Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video
classiﬁcation with convolutional neural networks. In CVPR ,
2014. 2
[28] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang,

2. Layer Normalization


2014. 2
[28] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang,
Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola,
Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. In arXiv preprint
arXiv:1705.06950 , 2017. 6
[29] Thomas N Kipf and Max Welling. Semi-supervised classiﬁcation with graph convolutional networks. In ICLR , 2017. 2,
3, 5
[30] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson,
Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome:
Connecting language and vision using crowdsourced dense
image annotations. IJCV , 123(1), 2017. 3
[31] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan

2. Layer Normalization


Popov, Matteo Malloci, Tom Duerig, et al. The open images dataset v4: Uniﬁed image classiﬁcation, object detection, and visual relationship detection at scale. IJCV , 2020.
4, 8
[32] Ivan Laptev. On space-time interest points. IJCV , 64(2-3),
2005. 2
[33] Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao’ou
Tang. Vip-cnn: Visual phrase guided convolutional neural
network. In CVPR , 2017. 3
[34] Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang. Scene graph generation from objects, phrases
and region captions. In ICCV , 2017. 7
[35] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard
Zemel. Gated graph sequence neural networks. In ICLR ,
2016. 5
[36] Yixuan Li, Zixu Wang, Limin Wang, and Gangshan Wu. Actions as moving points. In ECCV , 2020. 8
[37] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,

2. Layer Normalization


[37] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence
Zitnick. Microsoft coco: Common objects in context. In
ECCV , 2014. 12
[38] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian
Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C
Berg. Ssd: Single shot multibox detector. In ECCV , 2016. 2
[39] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li FeiFei. Visual relationship detection with language priors. In
ECCV , 2016. 6
[40] Chih-Yao Ma, Asim Kadav, Iain Melvin, Zsolt Kira, Ghassan
AlRegib, and Hans Peter Graf. Attend and interact: Higher-order object interactions for video understanding. In CVPR ,
2018. 2
[41] Marcin Marszalek, Ivan Laptev, and Cordelia Schmid. Actions in Context. In CVPR , 2009. 2
[42] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu,

2. Layer Normalization


[42] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu,
Xiaolong Wang, and Trevor Darrell. Something-else: Compositional action recognition with spatial-temporal interaction networks. In CVPR , 2020. 2
[43] Aude Oliva and Antonio Torralba. The role of context in object recognition. Trends in cognitive sciences , 11(12), 2007.
2
[44] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen,
and Song-Chun Zhu. Learning human-object interactions by
graph parsing neural networks. In ECCV , 2018. 2
[45] Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos,
Eric Wiewiora, and Serge Belongie. Objects in context. In
ICCV , 2007. 2
[46] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region
proposal networks. In NeurIPS , 2015. 1, 2, 3, 4, 12
[47] Adam Santoro, David Raposo, David G Barrett, Mateusz
Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy

2. Layer Normalization


[47] Adam Santoro, David Raposo, David G Barrett, Mateusz
Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy
Lillicrap. A simple neural network module for relational reasoning. In NeurIPS , 2017. 5
[48] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks , 20(1),
2008. 2, 5
[49] Gunnar A. Sigurdsson, G. Varol, X. Wang, Ali Farhadi, I.
Laptev, and A. Gupta. Hollywood in homes: Crowdsourcing
data collection for activity understanding. In ECCV , 2016.
1, 3, 6
[50] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In
NeurIPS , 2014. 2
[51] Gurkirt Singh, Suman Saha, Michael Sapienza, Philip HS
Torr, and Fabio Cuzzolin. Online real-time multiple spatiotemporal action localisation and prediction. In ICCV ,
2017. 2, 8

2. Layer Normalization


Torr, and Fabio Cuzzolin. Online real-time multiple spatiotemporal action localisation and prediction. In ICCV ,
2017. 2, 8
[52] Lin Song, Shiwei Zhang, Gang Yu, and Hongbin Sun. Tacnet: Transition-aware context network for spatio-temporal
action detection. In CVPR , 2019. 8
[53] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.
Ucf101: A dataset of 101 human actions classes from videos
in the wild. In arXiv preprint arXiv:1212.0402 , 2012. 2, 6,
8, 12
[54] Chen Sun, Abhinav Shrivastava, Carl V ondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric
relation network. In ECCV , 2018. 1, 2, 3, 4, 5, 8
[55] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann
LeCun, and Manohar Paluri. A closer look at spatiotemporal
convolutions for action recognition. In CVPR , 2018. 2

2. Layer Normalization


LeCun, and Manohar Paluri. A closer look at spatiotemporal
convolutions for action recognition. In CVPR , 2018. 2
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In NeurIPS , 2017. 2,
5, 6, 13
[57] Petar Veli ˇckovi ´c, Guillem Cucurull, Arantxa Casanova,
Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR , 2018. 3, 5, 6, 12
[58] Heng Wang, Alexander Kl ¨aser, Cordelia Schmid, and
Cheng-Lin Liu. Dense trajectories and motion boundary descriptors for action recognition. IJCV , 103(1), 2013. 2
[59] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR , 2018. 2, 5, 6,
12
[60] Xiaolong Wang and Abhinav Gupta. Videos as space-time

2. Layer Normalization


12
[60] Xiaolong Wang and Abhinav Gupta. Videos as space-time
region graphs. In ECCV , 2018. 1, 2, 3, 4, 5, 8
[61] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term
feature banks for detailed video understanding. In CVPR ,
2019. 1, 2, 3, 5, 6, 7, 8, 12
[62] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin,
and Michael Auli. Pay less attention with lightweight and
dynamic convolutions. In ICLR , 2019. 2
[63] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen
Lo, and Ross Girshick. Detectron2. https://github.
com/facebookresearch/detectron2 , 2019. 12
[64] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei.
Scene graph generation by iterative message passing. In
CVPR , 2017. 3, 7
[65] Huijuan Xu, Lizhi Yang, Stan Sclaroff, Kate Saenko, and

2. Layer Normalization


CVPR , 2017. 3, 7
[65] Huijuan Xu, Lizhi Yang, Stan Sclaroff, Kate Saenko, and
Trevor Darrell. Spatio-temporal action detection with multiobject interaction. In arXiv preprint arXiv:2004.00180 ,
2020. 2
[66] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi
Parikh. Graph r-cnn for scene graph generation. In ECCV ,
2018. 3
[67] Xitong Yang, Xiaodong Yang, Ming-Yu Liu, Fanyi Xiao,
Larry S Davis, and Jan Kautz. Step: Spatio-temporal progressive learning for video action detection. In CVPR , 2019.
8
[68] Bangpeng Yao and Li Fei-Fei. Modeling mutual context of
object and human pose in human-object interaction activities. In CVPR , 2010. 2
[69] Ji Zhang, Kevin J Shih, Ahmed Elgammal, Andrew Tao,
and Bryan Catanzaro. Graphical contrastive losses for scene
graph parsing. In CVPR , 2019. 7
[70] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr. Dynamic graph message passing networks. In CVPR , 2020. 2

2. Layer Normalization


[70] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr. Dynamic graph message passing networks. In CVPR , 2020. 2
[71] Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia
Schmid. A structured model for action detection. In CVPR ,
2019. 1, 2, 3, 5, 8
[72] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV ,
2018. 2
Appendix
In this appendix, we additional analysis on A V A (Sec. A)
and implementation details (Sec. B).
A. A V A per-class analysis
Figure 7 presents per-class results on A V A for our SlowFast baseline, spatial and spatio-temporal graph models
when using a ResNet-50 backbone. We additionally show
the number of examples in the training set. Our proposed
graph models improve on both head and tail classes in A V A.
In fact, the largest absolute improvements are observed on
tail classes such as “cut”, “swim” and “sing to”. However,
note that the absolute accuracy of each class is still correlated with the number of training examples. The classes
which all variants of our model perform the worst on, are

2. Layer Normalization


note that the absolute accuracy of each class is still correlated with the number of training examples. The classes
which all variants of our model perform the worst on, are
the classes with the least training examples such as “point
to (an object)”, “take a photo” and “turn (eg a screwdriver)”.
To emphasise the long-tailed distribution of A V A, note that
“point to (an object)” has the fewest training examples (just
97), whilst “watch (a person)” has the most with 168 148.
B. Additional Implementation details
Person detections For our spatio-temporal action detection experiments, we use external person detections as our
actor region proposals like [11, 61]. For our experiments
on A V A, we use the person detections publicly released
by [11, 61] during both training and testing. More speciﬁcally, this is a Faster-RCNN [46] model, pretrained on Microsoft COCO [37] and then ﬁnetuned on the training set of
A V A [17], using Detectron [63].
For our experiments on UCF101-24 [53], we ﬁnetune a Faster-RCNN detector pretrained on COCO on the
UCF101-24 training set using Detectron.

2. Layer Normalization


UCF101-24 training set using Detectron.
When training on both A V A and UCF101-24 datasets, we
use both ground-truth and predicted person bounding boxes.
Predicted bounding boxes are assigned labels by matching
them to ground-truth boxes using an IoU threshold of 0.75.
Predicted bounding boxes which do not match any ground
truth boxes act as negative examples for all action classes.
OpenImages object detector For our “explicit object
representation” experiments on A V A, we use the publicly
available Faster-RCNN detector trained on OpenImages
v41.
Optimiser hyperparameters We train our models following the settings of the publicly released SlowFast
code [11]. We train for 20 epochs using synchronous
Stochastic Gradient Descent (SGD) and a momentum of 0.9
1https://tfhub.dev/google/faster_rcnn/
openimages_v4/inception_resnet_v2/1on 8 Nvidia V100 (16 GB) GPUs. The learning rate was set
to 0.1, and reduced by a factor of 10 after 10 and 15 epochs
respectively. We employed a linear-warmup schedule for
the learning rate, increasing linearly from 1:25104to0:1

2. Layer Normalization


respectively. We employed a linear-warmup schedule for
the learning rate, increasing linearly from 1:25104to0:1
in the ﬁrst 5 epochs. We also used a weight decay of 107.
An epoch is deﬁned as all the keyframes in the dataset.
Training loss functions For spatio-temporal action
recognition on A V A and UCF101-24, we use the binary
cross-entropy loss function. The loss function for a single
example is:
L(x;y) =1
CCX
iyilog ((xi))+(1yi) log (1(xi));
(10)
whereCis the number of classes, is the sigmoid activation function, x2RCdenotes the logits predicted by the
network and yi2f0;1gdenotes the binary ground truth for
theithclass.
For scene graph classiﬁcation, we use two loss functions:
One for predicting the object class of each bounding box
proposal, and another for predicting the relationship label
between each of theN(N1)
2pairs of object proposals. For
the former, we use the softmax cross-entropy, as each proposal can only be assigned a single object class. And for

2. Layer Normalization


2pairs of object proposals. For
the former, we use the softmax cross-entropy, as each proposal can only be assigned a single object class. And for
the latter, we use the sigmoid cross-entropy, as there can
be multiple relationship labels between any pair of objects.
Concretely, the loss is
L(x;w;y;z ) =Lobject(x;y) +Lrel(w;z) (11)
Lobject(x;y) =1
NNX
iCX
jyijlog (Softmax (x)ij)(12)
Lrel(w;z) =2
N(N1)RNX
ii1X
jRX
rzijrlog ((wijr))
+ (1zijr) log (1(wijr)): (13)
Here,Nis the number of object proposals in the video clip,
Rthe number of relationship classes and Cthe number of
object classes. yis the one-hot, ground-truth object class label,zthe binary ground-truth relationship label, and xand
wdenote the object- and relationship-logits respectively.
On Action Genome [23], there are R= 25 relationship

2. Layer Normalization


wdenote the object- and relationship-logits respectively.
On Action Genome [23], there are R= 25 relationship
classes, and C= 35 object classes. We set = 0:5during
training on Action Genome to prevent the object class loss
from dominating the overall loss (the softmax cross entropy
has a higher loss than the binary sigmoid cross entropy at
the start of training).
Multi-headed attention For the GAT [57] and NonLocal [59] message-passing functions used in the paper
Mean AP
Number of training examples
0.000.250.500.751.00
100100010000100000watch (a person)
stand
talk to (e.g. self a person a
listen to (a person)
sit
carry/hold (an object)
walk
touch (an object)
bend/bow (at the waist)
lie/sleep
ride (e.g. a bike a car a horse)
run/jog
answer phone
dance
eat
smoke
fight/hit (a person)
drink
crouch/kneel
read
martial art
grab (a person)
watch (e.g. TV)
sing to (e.g. self a person a
play musical instrument

2. Layer Normalization


read
martial art
grab (a person)
watch (e.g. TV)
sing to (e.g. self a person a
play musical instrument
open (e.g. a window a car
drive (e.g. a car a truck)
hand clap
get up
hug (a person)
give/serve (an object) to (a
close (e.g. a door a box)
write
sail boat
kiss (a person)
listen (e.g. to music)
hand shake
take (an object) from (a
put down
lift/pick up
text on/look at a cellphone
lift (a person)
push (an object)
push (another person)
hand wave
pull (an object)
dress/put on clothing
fall down
climb (e.g. a mountain)
throw
jump/leap
enter
shoot
cut
take a photo
hit (an object)
work on a computer
turn (e.g. a screwdriver)
swim
point to (an object)
Baseline Spatial Spatio-temporal Training examplesFigure 7: Mean Average Precision for each action class on A V A, for the SlowFast baseline, spatial- and spatio-temporal graph models with

2. Layer Normalization


a ResNet-50 backbone. The number of examples in the training set are shown with the blue line on the right, vertical axis. Note that a
logarithmic scale is used, as A V A has a long-tailed distribution of labels.
Table 6: Ablation study of the number of heads when using GAT
as the message passing function. Mean AP reported on A V A using
a ResNet-50 backbone.
Number of heads 1 2 3 4 5
Mean AP 25.4 25.8 25.9 25.9 25.9
(Sec 3.4), it is common to compute “multi-headed” attention [56]. In all experiments reported in the main paper,
we used 4 heads, as motivated by the ablation study using
GAT on the A V A dataset in Tab. 6. To combine the messages from each head, we performed an attention-weighted
convex combination as done in Eq. 9.
Multiscale testing We use three scales when performing
multiscale testing (as done in Table 3 of the main paper).
Speciﬁcally, we resize input video-clips so that the shortest
side is 224, 256 and 320 respectively.

2. Layer Normalization


ViViT: A Video Vision Transformer
Anurag Arnab*Mostafa Dehghani*Georg Heigold Chen Sun Mario Lu ˇci´c†Cordelia Schmid†
Google Research
faarnab, dehghani, heigold, chensun, lucic, cordelias g@google.com
Abstract
We present pure-transformer based models for video
classiﬁcation, drawing upon the recent success of such models in image classiﬁcation. Our model extracts spatiotemporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we
propose several, efﬁcient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show
how we can effectively regularise the model during training
and leverage pretrained image models to be able to train on
comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple
video classiﬁcation benchmarks including Kinetics 400 and
600, Epic Kitchens, Something-Something v2 and Moments
in Time, outperforming prior methods based on deep 3D

ViViT: A Video Vision Transformer

Summary:
We present pure-transformer based models for video classification, drawing
upon the recent success of such models in image classification. Our model
extracts spatio-temporal tokens from the input video, which are then encoded by
a series of transformer layers. In order to handle the long sequences of tokens
encountered in video, we propose several, efficient variants of our model which
factorise the spatial- and temporal-dimensions of the input. Although
transformer-based models are known to only be effective when large training
datasets are available, we show how we can effectively regularise the model
during training and leverage pretrained image models to be able to train on
comparatively small datasets. We conduct thorough ablation studies, and achieve
state-of-the-art results on multiple video classification benchmarks including
Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in
Time, outperforming prior methods based on deep 3D convolutional networks. To
facilitate further research, we release code at
https://github.com/google-research/scenic/tree/main/scenic/projects/vivit

Anurag Arnab
Mostafa Dehghani
Georg Heigold
Chen Sun
Mario Lučić
Cordelia Schmid

ICCV 2021. Code at
  https://github.com/google-research/scenic/tree/main/scenic/projects/vivit

1. Unified Graph Structured Models for Video Understanding

2. Layer Normalization

3. Attention Augmented Convolutional Networks

Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, Quoc V Le

4. Is Space-Time Attention All You Need for Video Understanding?

Gedas Bertasius, Heng Wang, Lorenzo Torresani

5. Language Models are Few-Shot Learners


video classiﬁcation benchmarks including Kinetics 400 and
600, Epic Kitchens, Something-Something v2 and Moments
in Time, outperforming prior methods based on deep 3D
convolutional networks. To facilitate further research, we
release code at https://github.com/google-research/scenic.
1. Introduction
Approaches based on deep convolutional neural networks have advanced the state-of-the-art across many standard datasets for vision problems since AlexNet [38]. At
the same time, the most prominent architecture of choice in
sequence-to-sequence modelling (e.g. in natural language
processing) is the transformer [68], which does not use convolutions, but is based on multi-headed self-attention. This
operation is particularly effective at modelling long-range
dependencies and allows the model to attend over all elements in the input sequence. This is in stark contrast to
convolutions where the corresponding “receptive ﬁeld” is
limited, and grows linearly with the depth of the network.
The success of attention-based models in NLP has recently inspired approaches in computer vision to integrate
transformers into CNNs [75, 7], as well as some attempts to
replace convolutions completely [49, 3, 53]. However, it is
*Equal contribution

ViViT: A Video Vision Transformer

2. Layer Normalization


transformers into CNNs [75, 7], as well as some attempts to
replace convolutions completely [49, 3, 53]. However, it is
*Equal contribution
†Equal advisingonly very recently with the Vision Transformer (ViT) [18],
that a pure-transformer based architecture has outperformed
its convolutional counterparts in image classiﬁcation. Dosovitskiy et al. [18] closely followed the original transformer
architecture of [68], and noticed that its main beneﬁts
were observed at large scale – as transformers lack some
of the inductive biases of convolutions (such as translational equivariance), they seem to require more data [18]
or stronger regularisation [64].
Inspired by ViT, and the fact that attention-based architectures are an intuitive choice for modelling longrange contextual relationships in video, we develop several transformer-based models for video classiﬁcation. Currently, the most performant models are based on deep 3D
convolutional architectures [8, 20, 21] which were a natural extension of image classiﬁcation CNNs [27, 60]. Recently, these models were augmented by incorporating selfattention into their later layers to better capture long-range
dependencies [75, 23, 79, 1].

ViViT: A Video Vision Transformer

2. Layer Normalization


dependencies [75, 23, 79, 1].
As shown in Fig. 1, we propose pure-transformer models for video classiﬁcation. The main operation performed
in this architecture is self-attention, and it is computed on
a sequence of spatio-temporal tokens that we extract from
the input video. To effectively process the large number of
spatio-temporal tokens that may be encountered in video,
we present several methods of factorising our model along
spatial and temporal dimensions to increase efﬁciency and
scalability. Furthermore, to train our model effectively on
smaller datasets, we show how to reguliarise our model during training and leverage pretrained image models.
We also note that convolutional models have been developed by the community for several years, and there are
thus many “best practices” associated with such models.
As pure-transformer models present different characteristics, we need to determine the best design choices for such
architectures. We conduct a thorough ablation analysis of
tokenisation strategies, model architecture and regularisation methods. Informed by this analysis, we achieve stateof-the-art results on multiple standard video classiﬁcation
benchmarks, including Kinetics 400 and 600 [35], Epic

ViViT: A Video Vision Transformer

2. Layer Normalization


benchmarks, including Kinetics 400 and 600 [35], Epic
Kitchens 100 [13], Something-Something v2 [26] and Moments in Time [45].arXiv:2103.15691v2  [cs.CV]  1 Nov 2021
…1CLS032NPosition + Token EmbeddingMLP HeadClassFactorisedEncoderL×
K         V         Q Self-AttentionTransformer  Encoder MLPLayer Norm
Layer NormMulti-HeadDot-Product Attention
Embed to tokensFactorisedSelf-Attention
21NFactorisedDot-Product
●●●SpatialSpatial●●●TemporalTemporal
SpatialTemporalSpatialTemporal●●●SpatialTemporal●●●FuseSpatialTemporalFuse
●●●21N●●●21N●●●Figure 1: We propose a pure-transformer architecture for video classiﬁcation, inspired by the recent success of such models for images [18].
To effectively process a large number of spatio-temporal tokens, we develop several model variants which factorise different components
of the transformer encoder over the spatial- and temporal-dimensions. As shown on the right, these factorisations correspond to different
attention patterns over space and time.
2. Related Work

ViViT: A Video Vision Transformer

2. Layer Normalization