Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised
  Audio-Visual Video Parsing
Jie Fu Junyu Gao Changsheng Xu
Weakly-supervised audio-visual video parsing (WS-AVVP) aims to localize the
temporal extents of audio, visual and audio-visual event instances as well as
identify the corresponding event categories with only video-level category
labels for training. Most previous methods pay much attention to refining the
supervision for each modality or extracting fruitful cross-modality information
for more reliable feature learning. None of them have noticed the imbalanced
feature learning between different modalities in the task. In this paper, to
balance the feature learning processes of different modalities, a dynamic
gradient modulation (DGM) mechanism is explored, where a novel and effective
metric function is designed to measure the imbalanced feature learning between
audio and visual modalities. Furthermore, principle analysis indicates that the
multimodal confusing calculation will hamper the precise measurement of
multimodal imbalanced feature learning, which further weakens the effectiveness
of our DGM mechanism. To cope with this issue, a modality-separated decision
unit (MSDU) is designed for more precise measurement of imbalanced feature
learning between audio and visual modalities. Comprehensive experiments are
conducted on public benchmarks and the corresponding experimental results
demonstrate the effectiveness of our proposed method.