Language-Universal Phonetic Representation in Multilingual Speech
  Pretraining for Low-Resource Speech Recognition
Siyuan Feng Ming Tu Rui Xia Chuanzeng Huang Yuxuan Wang
We improve low-resource ASR by integrating the ideas of multilingual training
and self-supervised learning. Concretely, we leverage an International Phonetic
Alphabet (IPA) multilingual model to create frame-level pseudo labels for
unlabeled speech, and use these pseudo labels to guide hidden-unit BERT
(HuBERT) based speech pretraining in a phonetically-informed manner. The
experiments on the Multilingual Speech (MLS) Corpus show that the proposed
approach consistently outperforms the standard HuBERT on all the target
languages. Moreover, on 3 of the 4 languages, comparing to the standard HuBERT,
the approach performs better, meanwhile is able to save supervised training
data by 1.5k hours (75%) at most. Our approach outperforms most of the state of
the arts, with much less pretraining data in terms of hours and language
diversity. Compared to XLSR-53 and a retraining based multilingual method, our
approach performs better with full and limited finetuning data scenarios.