Rethink Cross-Modal Fusion in Weakly-Supervised Audio-Visual Video
  Parsing
Yating Xu Conghui Hu Gim Hee Lee
Existing works on weakly-supervised audio-visual video parsing adopt hybrid
attention network (HAN) as the multi-modal embedding to capture the cross-modal
context. It embeds the audio and visual modalities with a shared network, where
the cross-attention is performed at the input. However, such an early fusion
method highly entangles the two non-fully correlated modalities and leads to
sub-optimal performance in detecting single-modality events. To deal with this
problem, we propose the messenger-guided mid-fusion transformer to reduce the
uncorrelated cross-modal context in the fusion. The messengers condense the
full cross-modal context into a compact representation to only preserve useful
cross-modal information. Furthermore, due to the fact that microphones capture
audio events from all directions, while cameras only record visual events
within a restricted field of view, there is a more frequent occurrence of
unaligned cross-modal context from audio for visual event predictions. We thus
propose cross-audio prediction consistency to suppress the impact of irrelevant
audio information on visual event prediction. Experiments consistently
illustrate the superior performance of our framework compared to existing
state-of-the-art methods.