Bass Accompaniment Generation via Latent Diffusion
Marco Pasini Maarten Grachten Stefan Lattner
The ability to automatically generate music that appropriately matches an
arbitrary input track is a challenging task. We present a novel controllable
system for generating single stems to accompany musical mixes of arbitrary
length. At the core of our method are audio autoencoders that efficiently
compress audio waveform samples into invertible latent representations, and a
conditional latent diffusion model that takes as input the latent encoding of a
mix and generates the latent encoding of a corresponding stem. To provide
control over the timbre of generated samples, we introduce a technique to
ground the latent space to a user-provided reference style during diffusion
sampling. For further improving audio quality, we adapt classifier-free
guidance to avoid distortions at high guidance strengths when generating an
unbounded latent space. We train our model on a dataset of pairs of mixes and
matching bass stems. Quantitative experiments demonstrate that, given an input
mix, the proposed system can generate basslines with user-specified timbres.
Our controllable conditional audio generation framework represents a
significant step forward in creating generative AI tools to assist musicians in
music production.