EnCodecMAE: Leveraging neural codecs for universal audio representation
  learning
Leonardo Pepino Pablo Riera Luciana Ferrer
The goal of universal audio representation learning is to obtain foundational
models that can be used for a variety of downstream tasks involving speech,
music or environmental sounds. To approach this problem, methods inspired by
self-supervised models from NLP, like BERT, are often used and adapted to
audio. These models rely on the discrete nature of text, hence adopting this
type of approach for audio processing requires either a change in the learning
objective or mapping the audio signal to a set of discrete classes. In this
work, we explore the use of EnCodec, a neural audio codec, to generate discrete
targets for learning an universal audio model based on a masked autoencoder
(MAE). We evaluate this approach, which we call EncodecMAE, on a wide range of
audio tasks spanning speech, music and environmental sounds, achieving
performances comparable or better than leading audio representation models.