An Integration of Pre-Trained Speech and Language Models for End-to-End
  Speech Recognition
Yukiya Hono Koh Mitsuda Tianyu Zhao Kentaro Mitsui Toshiaki Wakatsuki Kei Sawada
Advances in machine learning have made it possible to perform various text
and speech processing tasks, including automatic speech recognition (ASR), in
an end-to-end (E2E) manner. Since typical E2E approaches require large amounts
of training data and resources, leveraging pre-trained foundation models
instead of training from scratch is gaining attention. Although there have been
attempts to use pre-trained speech and language models in ASR, most of them are
limited to using either. This paper explores the potential of integrating a
pre-trained speech representation model with a large language model (LLM) for
E2E ASR. The proposed model enables E2E ASR by generating text tokens in an
autoregressive manner via speech representations as speech prompts, taking
advantage of the vast knowledge provided by the LLM. Furthermore, the proposed
model can incorporate remarkable developments for LLM utilization, such as
inference optimization and parameter-efficient domain adaptation. Experimental
results show that the proposed model achieves performance comparable to modern
E2E ASR models.