A Real-Time Active Speaker Detection System Integrating an Audio-Visual
  Signal with a Spatial Querying Mechanism
Ilya Gurvich Ido Leichter Dharmendar Reddy Palle Yossi Asher Alon Vinnikov Igor Abramovski Vishak Gopal Ross Cutler Eyal Krupka
We introduce a distinctive real-time, causal, neural network-based active
speaker detection system optimized for low-power edge computing. This system
drives a virtual cinematography module and is deployed on a commercial device.
The system uses data originating from a microphone array and a 360-degree
camera. Our network requires only 127 MFLOPs per participant, for a meeting
with 14 participants. Unlike previous work, we examine the error rate of our
network when the computational budget is exhausted, and find that it exhibits
graceful degradation, allowing the system to operate reasonably well even in
this case. Departing from conventional DOA estimation approaches, our network
learns to query the available acoustic data, considering the detected head
locations. We train and evaluate our algorithm on a realistic meetings dataset
featuring up to 14 participants in the same meeting, overlapped speech, and
other challenging scenarios.