Performance Conditioning for Diffusion-Based Multi-Instrument Music
  Synthesis
Ben Maman Johannes Zeitler Meinard Müller Amit H. Bermano
Generating multi-instrument music from symbolic music representations is an
important task in Music Information Retrieval (MIR). A central but still
largely unsolved problem in this context is musically and acoustically informed
control in the generation process. As the main contribution of this work, we
propose enhancing control of multi-instrument synthesis by conditioning a
generative model on a specific performance and recording environment, thus
allowing for better guidance of timbre and style. Building on state-of-the-art
diffusion-based music generative models, we introduce performance conditioning
- a simple tool indicating the generative model to synthesize music with style
and timbre of specific instruments taken from specific performances. Our
prototype is evaluated using uncurated performances with diverse
instrumentation and achieves state-of-the-art FAD realism scores while allowing
novel timbre and style control. Our project page, including samples and
demonstrations, is available at benadar293.github.io/midipm