Audio-Visual Instance Segmentation
Ruohao Guo Yaru Chen Yanyu Qi Wenzhen Yue Dantong Niu Xianghua Ying
In this paper, we propose a new multi-modal task, namely audio-visual
instance segmentation (AVIS), in which the goal is to identify, segment, and
track individual sounding object instances in audible videos, simultaneously.
To our knowledge, it is the first time that instance segmentation has been
extended into the audio-visual domain. To better facilitate this research, we
construct the first audio-visual instance segmentation benchmark (AVISeg).
Specifically, AVISeg consists of 1,258 videos with an average duration of 62.6
seconds from YouTube and public audio-visual datasets, where 117 videos have
been annotated by using an interactive semi-automatic labeling tool based on
the Segment Anything Model (SAM). In addition, we present a simple baseline
model for the AVIS task. Our new model introduces an audio branch and a
cross-modal fusion module to Mask2Former to locate all sounding objects.
Finally, we evaluate the proposed method using two backbones on AVISeg. We
believe that AVIS will inspire the community towards a more comprehensive
multi-modal understanding.