Weakly-Supervised Audio-Visual Segmentation
Shentong Mo Bhiksha Raj
Audio-visual segmentation is a challenging task that aims to predict
pixel-level masks for sound sources in a video. Previous work applied a
comprehensive manually designed architecture with countless pixel-wise accurate
masks as supervision. However, these pixel-level masks are expensive and not
available in all cases. In this work, we aim to simplify the supervision as the
instance-level annotation, i.e., weakly-supervised audio-visual segmentation.
We present a novel Weakly-Supervised Audio-Visual Segmentation framework,
namely WS-AVS, that can learn multi-scale audio-visual alignment with
multi-scale multiple-instance contrastive learning for audio-visual
segmentation. Extensive experiments on AVSBench demonstrate the effectiveness
of our WS-AVS in the weakly-supervised audio-visual segmentation of
single-source and multi-source scenarios.