CM-PIE: Cross-modal perception for interactive-enhanced audio-visual
  video parsing
Yaru Chen Ruohao Guo Xubo Liu Peipei Wu Guangyao Li Zhenbo Li Wenwu Wang
Audio-visual video parsing is the task of categorizing a video at the segment
level with weak labels, and predicting them as audible or visible events.
Recent methods for this task leverage the attention mechanism to capture the
semantic correlations among the whole video across the audio-visual modalities.
However, these approaches have overlooked the importance of individual segments
within a video and the relationship among them, and tend to rely on a single
modality when learning features. In this paper, we propose a novel
interactive-enhanced cross-modal perception method~(CM-PIE), which can learn
fine-grained features by applying a segment-based attention module.
Furthermore, a cross-modal aggregation block is introduced to jointly optimize
the semantic representation of audio and visual signals by enhancing
inter-modal interactions. The experimental results show that our model offers
improved parsing performance on the Look, Listen, and Parse dataset compared to
other methods.