Leveraging Multilingual Self-Supervised Pretrained Models for
  Sequence-to-Sequence End-to-End Spoken Language Understanding
Pavel Denisov Ngoc Thang Vu
A number of methods have been proposed for End-to-End Spoken Language
Understanding (E2E-SLU) using pretrained models, however their evaluation often
lacks multilingual setup and tasks that require prediction of lexical fillers,
such as slot filling. In this work, we propose a unified method that integrates
multilingual pretrained speech and text models and performs E2E-SLU on six
datasets in four languages in a generative manner, including the prediction of
lexical fillers. We investigate how the proposed method can be improved by
pretraining on widely available speech recognition data using several training
objectives. Pretraining on 7000 hours of multilingual data allows us to
outperform the state-of-the-art ultimately on two SLU datasets and partly on
two more SLU datasets. Finally, we examine the cross-lingual capabilities of
the proposed model and improve on the best known result on the
PortMEDIA-Language dataset by almost half, achieving a Concept/Value Error Rate
of 23.65%.