Learning Human Action Recognition Representations Without Real Humans
Howard Zhong Samarth Mishra Donghyun Kim SouYoung Jin Rameswar Panda Hilde Kuehne Leonid Karlinsky Venkatesh Saligrama Aude Oliva Rogerio Feris
Pre-training on massive video datasets has become essential to achieve high
action recognition performance on smaller downstream datasets. However, most
large-scale video datasets contain images of people and hence are accompanied
with issues related to privacy, ethics, and data protection, often preventing
them from being publicly shared for reproducible research. Existing work has
attempted to alleviate these problems by blurring faces, downsampling videos,
or training on synthetic data. On the other hand, analysis on the
transferability of privacy-preserving pre-trained models to downstream tasks
has been limited. In this work, we study this problem by first asking the
question: can we pre-train models for human action recognition with data that
does not include real humans? To this end, we present, for the first time, a
benchmark that leverages real-world videos with humans removed and synthetic
data containing virtual humans to pre-train a model. We then evaluate the
transferability of the representation learned on this data to a diverse set of
downstream action recognition benchmarks. Furthermore, we propose a novel
pre-training strategy, called Privacy-Preserving MAE-Align, to effectively
combine synthetic data and human-removed real data. Our approach outperforms
previous baselines by up to 5% and closes the performance gap between human and
no-human action recognition representations on downstream tasks, for both
linear probing and fine-tuning. Our benchmark, code, and models are available
at https://github.com/howardzh01/PPMA .