Exploring how a Generative AI interprets music
Gabriela Barenboim Luigi Del Debbio Johannes Hirn Veronica Sanz
We use Google's MusicVAE, a Variational Auto-Encoder with a 512-dimensional
latent space to represent a few bars of music, and organize the latent
dimensions according to their relevance in describing music. We find that, on
average, most latent neurons remain silent when fed real music tracks: we call
these "noise" neurons. The remaining few dozens of latent neurons that do fire
are called "music neurons". We ask which neurons carry the musical information
and what kind of musical information they encode, namely something that can be
identified as pitch, rhythm or melody. We find that most of the information
about pitch and rhythm is encoded in the first few music neurons: the neural
network has thus constructed a couple of variables that non-linearly encode
many human-defined variables used to describe pitch and rhythm. The concept of
melody only seems to show up in independent neurons for longer sequences of
music.