DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion
  Models
Chao Huang Susan Liang Yapeng Tian Anurag Kumar Chenliang Xu
We propose DAVIS, a Diffusion model-based Audio-VIusal Separation framework
that solves the audio-visual sound source separation task through a generative
manner. While existing discriminative methods that perform mask regression have
made remarkable progress in this field, they face limitations in capturing the
complex data distribution required for high-quality separation of sounds from
diverse categories. In contrast, DAVIS leverages a generative diffusion model
and a Separation U-Net to synthesize separated magnitudes starting from
Gaussian noises, conditioned on both the audio mixture and the visual footage.
With its generative objective, DAVIS is better suited to achieving the goal of
high-quality sound separation across diverse categories. We compare DAVIS to
existing state-of-the-art discriminative audio-visual separation methods on the
domain-specific MUSIC dataset and the open-domain AVE dataset, and results show
that DAVIS outperforms other methods in separation quality, demonstrating the
advantages of our framework for tackling the audio-visual source separation
task.