Multitrack Music Transcription with a Time-Frequency Perceiver
Wei-Tsung Lu Ju-Chiang Wang Yun-Ning Hung
Multitrack music transcription aims to transcribe a music audio input into
the musical notes of multiple instruments simultaneously. It is a very
challenging task that typically requires a more complex model to achieve
satisfactory result. In addition, prior works mostly focus on transcriptions of
regular instruments, however, neglecting vocals, which are usually the most
important signal source if present in a piece of music. In this paper, we
propose a novel deep neural network architecture, Perceiver TF, to model the
time-frequency representation of audio input for multitrack transcription.
Perceiver TF augments the Perceiver architecture by introducing a hierarchical
expansion with an additional Transformer layer to model temporal coherence.
Accordingly, our model inherits the benefits of Perceiver that posses better
scalability, allowing it to well handle transcriptions of many instruments in a
single model. In experiments, we train a Perceiver TF to model 12 instrument
classes as well as vocal in a multi-task learning manner. Our result
demonstrates that the proposed system outperforms the state-of-the-art
counterparts (e.g., MT3 and SpecTNT) on various public datasets.