Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation
Jinxiang Liu Chen Ju Chaofan Ma Yanfeng Wang Yu Wang Ya Zhang
The goal of the audio-visual segmentation (AVS) task is to segment the
sounding objects in the video frames using audio cues. However, current
fusion-based methods have the performance limitations due to the small
receptive field of convolution and inadequate fusion of audio-visual features.
To overcome these issues, we propose a novel \textbf{Au}dio-aware
query-enhanced \textbf{TR}ansformer (AuTR) to tackle the task. Unlike existing
methods, our approach introduces a multimodal transformer architecture that
enables deep fusion and aggregation of audio-visual features. Furthermore, we
devise an audio-aware query-enhanced transformer decoder that explicitly helps
the model focus on the segmentation of the pinpointed sounding objects based on
audio signals, while disregarding silent yet salient objects. Experimental
results show that our method outperforms previous methods and demonstrates
better generalization ability in multi-sound and open-set scenarios.