Multimodal Variational Auto-encoder based Audio-Visual Segmentation
Yuxin Mao Jing Zhang Mochu Xiang Yiran Zhong Yuchao Dai
We propose an Explicit Conditional Multimodal Variational Auto-Encoder
(ECMVAE) for audio-visual segmentation (AVS), aiming to segment sound sources
in the video sequence. Existing AVS methods focus on implicit feature fusion
strategies, where models are trained to fit the discrete samples in the
dataset. With a limited and less diverse dataset, the resulting performance is
usually unsatisfactory. In contrast, we address this problem from an effective
representation learning perspective, aiming to model the contribution of each
modality explicitly. Specifically, we find that audio contains critical
category information of the sound producers, and visual data provides candidate
sound producer(s). Their shared information corresponds to the target sound
producer(s) shown in the visual data. In this case, cross-modal shared
representation learning is especially important for AVS. To achieve this, our
ECMVAE factorizes the representations of each modality with a modality-shared
representation and a modality-specific representation. An orthogonality
constraint is applied between the shared and specific representations to
maintain the exclusive attribute of the factorized latent code. Further, a
mutual information maximization regularizer is introduced to achieve extensive
exploration of each modality. Quantitative and qualitative evaluations on the
AVSBench demonstrate the effectiveness of our approach, leading to a new
state-of-the-art for AVS, with a 3.84 mIOU performance leap on the challenging
MS3 subset for multiple sound source segmentation.