SLM: Bridge the thin gap between speech and text foundation models
Mingqiu Wang Wei Han Izhak Shafran Zelin Wu Chung-Cheng Chiu Yuan Cao Yongqiang Wang Nanxin Chen Yu Zhang Hagen Soltau Paul Rubenstein Lukas Zilka Dian Yu Zhong Meng Golan Pundak Nikhil Siddhartha Johan Schalkwyk Yonghui Wu
We present a joint Speech and Language Model (SLM), a multitask,
multilingual, and dual-modal model that takes advantage of pretrained
foundational speech and language models. SLM freezes the pretrained foundation
models to maximally preserves their capabilities, and only trains a simple
adapter with just 1\% (156M) of the foundation models' parameters. This
adaptation not only leads SLM to achieve strong performance on conventional
tasks such as speech recognition (ASR) and speech translation (AST), but also
introduces the novel capability of zero-shot instruction-following for more
diverse tasks: given a speech input and a text instruction, SLM is able to
perform unseen generation tasks including contextual biasing ASR using
real-time context, dialog generation, speech continuation, and question
answering, etc. Our approach demonstrates that the representational gap between
pretrained speech and language models might be narrower than one would expect,
and can be bridged by a simple adaptation mechanism. As a result, SLM is not
only efficient to train, but also inherits strong capabilities already acquired
in foundation models of different modalities.