Teaching a Multilingual Large Language Model to Understand Multilingual
  Speech via Multi-Instructional Training
Pavel Denisov Ngoc Thang Vu
Recent advancements in language modeling have led to the emergence of Large
Language Models (LLMs) capable of various natural language processing tasks.
Despite their success in text-based tasks, applying LLMs to the speech domain
remains limited and challenging. This paper presents BLOOMZMMS, a novel model
that integrates a multilingual LLM with a multilingual speech encoder, aiming
to harness the capabilities of LLMs for speech recognition and beyond.
Utilizing a multi-instructional training approach, we demonstrate the
transferability of linguistic knowledge from the text to the speech modality.
Our experiments, conducted on 1900 hours of transcribed data from 139
languages, establish that a multilingual speech representation can be
effectively learned and aligned with a multilingual LLM. While this learned
representation initially shows limitations in task generalization, we address
this issue by generating synthetic targets in a multi-instructional style. Our
zero-shot evaluation results confirm the robustness of our approach across
multiple tasks, including speech translation and multilingual spoken language
understanding, thereby opening new avenues for applying LLMs in the speech
domain.