Efficiency-oriented approaches for self-supervised speech representation
  learning
Luis Lugo Valentin Vielzeuf
Self-supervised learning enables the training of large neural models without
the need for large, labeled datasets. It has been generating breakthroughs in
several fields, including computer vision, natural language processing,
biology, and speech. In particular, the state-of-the-art in several speech
processing applications, such as automatic speech recognition or speaker
identification, are models where the latent representation is learned using
self-supervised approaches. Several configurations exist in self-supervised
learning for speech, including contrastive, predictive, and multilingual
approaches. There is, however, a crucial limitation in most existing
approaches: their high computational costs. These costs limit the deployment of
models, the size of the training dataset, and the number of research groups
that can afford research with large self-supervised models. Likewise, we should
consider the environmental costs that high energy consumption implies. Efforts
in this direction comprise optimization of existing models, neural architecture
efficiency, improvements in finetuning for speech processing tasks, and data
efficiency. But despite current efforts, more work could be done to address
high computational costs in self-supervised representation learning.