REWIND Dataset: Privacy-preserving Speaking Status Segmentation from
  Multimodal Body Movement Signals in the Wild
Jose Vargas Quiros Chirag Raman Stephanie Tan Ekin Gedik Laura Cabrera-Quiros Hayley Hung
Recognizing speaking in humans is a central task towards understanding social
interactions. Ideally, speaking would be detected from individual voice
recordings, as done previously for meeting scenarios. However, individual voice
recordings are hard to obtain in the wild, especially in crowded mingling
scenarios due to cost, logistics, and privacy concerns. As an alternative,
machine learning models trained on video and wearable sensor data make it
possible to recognize speech by detecting its related gestures in an
unobtrusive, privacy-preserving way. These models themselves should ideally be
trained using labels obtained from the speech signal. However, existing
mingling datasets do not contain high quality audio recordings. Instead,
speaking status annotations have often been inferred by human annotators from
video, without validation of this approach against audio-based ground truth. In
this paper we revisit no-audio speaking status estimation by presenting the
first publicly available multimodal dataset with high-quality individual speech
recordings of 33 subjects in a professional networking event. We present three
baselines for no-audio speaking status segmentation: a) from video, b) from
body acceleration (chest-worn accelerometer), c) from body pose tracks. In all
cases we predict a 20Hz binary speaking status signal extracted from the audio,
a time resolution not available in previous datasets. In addition to providing
the signals and ground truth necessary to evaluate a wide range of speaking
status detection methods, the availability of audio in REWIND makes it suitable
for cross-modality studies not feasible with previous mingling datasets.
Finally, our flexible data consent setup creates new challenges for multimodal
systems under missing modalities.