Toward Fully Self-Supervised Multi-Pitch Estimation
Frank Cwitkowitz Zhiyao Duan
Multi-pitch estimation is a decades-long research problem involving the
detection of pitch activity associated with concurrent musical events within
multi-instrument mixtures. Supervised learning techniques have demonstrated
solid performance on more narrow characterizations of the task, but suffer from
limitations concerning the shortage of large-scale and diverse polyphonic music
datasets with multi-pitch annotations. We present a suite of self-supervised
learning objectives for multi-pitch estimation, which encourage the
concentration of support around harmonics, invariance to timbral
transformations, and equivariance to geometric transformations. These
objectives are sufficient to train an entirely convolutional autoencoder to
produce multi-pitch salience-grams directly, without any fine-tuning. Despite
training exclusively on a collection of synthetic single-note audio samples,
our fully self-supervised framework generalizes to polyphonic music mixtures,
and achieves performance comparable to supervised models trained on
conventional multi-pitch datasets.