TokenSplit: Using Discrete Speech Representations for Direct, Refined,
  and Transcript-Conditioned Speech Separation and Recognition
Hakan Erdogan Scott Wisdom Xuankai Chang Zalán Borsos Marco Tagliasacchi Neil Zeghidour John R. Hershey
We present TokenSplit, a speech separation model that acts on discrete token
sequences. The model is trained on multiple tasks simultaneously: separate and
transcribe each speech source, and generate speech from text. The model
operates on transcripts and audio token sequences and achieves multiple tasks
through masking of inputs. The model is a sequence-to-sequence encoder-decoder
model that uses the Transformer architecture. We also present a "refinement"
version of the model that predicts enhanced audio tokens from the audio tokens
of speech separated by a conventional separation model. Using both objective
metrics and subjective MUSHRA listening tests, we show that our model achieves
excellent performance in terms of separation, both with or without transcript
conditioning. We also measure the automatic speech recognition (ASR)
performance and provide audio samples of speech synthesis to demonstrate the
additional utility of our model.