Unified Modeling of Multi-Talker Overlapped Speech Recognition and
  Diarization with a Sidecar Separator
Lingwei Meng Jiawen Kang Mingyu Cui Haibin Wu Xixin Wu Helen Meng
Multi-talker overlapped speech poses a significant challenge for speech
recognition and diarization. Recent research indicated that these two tasks are
inter-dependent and complementary, motivating us to explore a unified modeling
method to address them in the context of overlapped speech. A recent study
proposed a cost-effective method to convert a single-talker automatic speech
recognition (ASR) system into a multi-talker one, by inserting a Sidecar
separator into the frozen well-trained ASR model. Extending on this, we
incorporate a diarization branch into the Sidecar, allowing for unified
modeling of both ASR and diarization with a negligible overhead of only 768
parameters. The proposed method yields better ASR results compared to the
baseline on LibriMix and LibriSpeechMix datasets. Moreover, without
sophisticated customization on the diarization task, our method achieves
acceptable diarization results on the two-speaker subset of CALLHOME with only
a few adaptation steps.