GestSync: Determining who is speaking without a talking head
Sindhu B Hegde Andrew Zisserman
In this paper we introduce a new synchronisation task, Gesture-Sync:
determining if a person's gestures are correlated with their speech or not. In
comparison to Lip-Sync, Gesture-Sync is far more challenging as there is a far
looser relationship between the voice and body movement than there is between
voice and lip motion. We introduce a dual-encoder model for this task, and
compare a number of input representations including RGB frames, keypoint
images, and keypoint vectors, assessing their performance and advantages. We
show that the model can be trained using self-supervised learning alone, and
evaluate its performance on the LRS3 dataset. Finally, we demonstrate
applications of Gesture-Sync for audio-visual synchronisation, and in
determining who is the speaker in a crowd, without seeing their faces. The
code, datasets and pre-trained models can be found at:
\url{https://www.robots.ox.ac.uk/~vgg/research/gestsync}.