XLS-R fine-tuning on noisy word boundaries for unsupervised speech
  segmentation into words
Robin Algayres Pablo Diego-Simon Benoit Sagot Emmanuel Dupoux
Due to the absence of explicit word boundaries in the speech stream, the task
of segmenting spoken sentences into word units without text supervision is
particularly challenging. In this work, we leverage the most recent
self-supervised speech models that have proved to quickly adapt to new tasks
through fine-tuning, even in low resource conditions. Taking inspiration from
semi-supervised learning, we fine-tune an XLS-R model to predict word
boundaries themselves produced by top-tier speech segmentation systems: DPDP,
VG-HuBERT, GradSeg and DP-Parse. Once XLS-R is fine-tuned, it is used to infer
new word boundary labels that are used in turn for another fine-tuning step.
Our method consistently improves the performance of each system and sets a new
state-of-the-art that is, on average 130% higher than the previous one as
measured by the F1 score on correctly discovered word tokens on five corpora
featuring different languages. Finally, our system can segment speech from
languages unseen during fine-tuning in a zero-shot fashion.