Speaker Mask Transformer for Multi-talker Overlapped Speech Recognition
Peng Shen Xugang Lu Hisashi Kawai
Multi-talker overlapped speech recognition remains a significant challenge,
requiring not only speech recognition but also speaker diarization tasks to be
addressed. In this paper, to better address these tasks, we first introduce
speaker labels into an autoregressive transformer-based speech recognition
model to support multi-speaker overlapped speech recognition. Then, to improve
speaker diarization, we propose a novel speaker mask branch to detection the
speech segments of individual speakers. With the proposed model, we can perform
both speech recognition and speaker diarization tasks simultaneously using a
single model. Experimental results on the LibriSpeech-based overlapped dataset
demonstrate the effectiveness of the proposed method in both speech recognition
and speaker diarization tasks, particularly enhancing the accuracy of speaker
diarization in relatively complex multi-talker scenarios.