Generative Context-aware Fine-tuning of Self-supervised Speech Models
Suwon Shon Kwangyoun Kim Prashant Sridhar Yi-Te Hsu Shinji Watanabe Karen Livescu
When performing tasks like automatic speech recognition or spoken language
understanding for a given utterance, access to preceding text or audio provides
contextual information can improve performance. Considering the recent advances
in generative large language models (LLM), we hypothesize that an LLM could
generate useful context information using the preceding text. With appropriate
prompts, LLM could generate a prediction of the next sentence or abstractive
text like titles or topics. In this paper, we study the use of LLM-generated
context information and propose an approach to distill the generated
information during fine-tuning of self-supervised speech models, which we refer
to as generative context-aware fine-tuning. This approach allows the fine-tuned
model to make improved predictions without access to the true surrounding
segments or to the LLM at inference time, while requiring only a very small
additional context module. We evaluate the proposed approach using the SLUE and
Libri-light benchmarks for several downstream tasks: automatic speech
recognition, named entity recognition, and sentiment analysis. The results show
that generative context-aware fine-tuning outperforms a context injection
fine-tuning approach that accesses the ground-truth previous text, and is
competitive with a generative context injection fine-tuning approach that
requires the LLM at inference time.