Leveraging Visual Supervision for Array-based Active Speaker Detection
  and Localization
Davide Berghi Philip J. B. Jackson
Conventional audio-visual approaches for active speaker detection (ASD)
typically rely on visually pre-extracted face tracks and the corresponding
single-channel audio to find the speaker in a video. Therefore, they tend to
fail every time the face of the speaker is not visible. We demonstrate that a
simple audio convolutional recurrent neural network (CRNN) trained with spatial
input features extracted from multichannel audio can perform simultaneous
horizontal active speaker detection and localization (ASDL), independently of
the visual modality. To address the time and cost of generating ground truth
labels to train such a system, we propose a new self-supervised training
pipeline that embraces a ``student-teacher'' learning approach. A conventional
pre-trained active speaker detector is adopted as a ``teacher'' network to
provide the position of the speakers as pseudo-labels. The multichannel audio
``student'' network is trained to generate the same results. At inference, the
student network can generalize and locate also the occluded speakers that the
teacher network is not able to detect visually, yielding considerable
improvements in recall rate. Experiments on the TragicTalkers dataset show that
an audio network trained with the proposed self-supervised learning approach
can exceed the performance of the typical audio-visual methods and produce
results competitive with the costly conventional supervised training. We
demonstrate that improvements can be achieved when minimal manual supervision
is introduced in the learning pipeline. Further gains may be sought with larger
training sets and integrating vision with the multichannel audio system.