Progressive Spatio-temporal Perception for Audio-Visual Question
  Answering
Guangyao Li Wenxuan Hou Di Hu
Audio-Visual Question Answering (AVQA) task aims to answer questions about
different visual objects, sounds, and their associations in videos. Such
naturally multi-modal videos are composed of rich and complex dynamic
audio-visual components, where most of which could be unrelated to the given
questions, or even play as interference in answering the content of interest.
Oppositely, only focusing on the question-aware audio-visual content could get
rid of influence, meanwhile enabling the model to answer more efficiently. In
this paper, we propose a Progressive Spatio-Temporal Perception Network
(PSTP-Net), which contains three modules that progressively identify key
spatio-temporal regions w.r.t. questions. Specifically, a temporal segment
selection module is first introduced to select the most relevant audio-visual
segments related to the given question. Then, a spatial region selection module
is utilized to choose the most relevant regions associated with the question
from the selected temporal segments. To further refine the selection of
features, an audio-guided visual attention module is employed to perceive the
association between auido and selected spatial regions. Finally, the
spatio-temporal features from these modules are integrated for answering the
question. Extensive experimental results on the public MUSIC-AVQA and AVQA
datasets provide compelling evidence of the effectiveness and efficiency of
PSTP-Net. Code is available at:
\href{https://github.com/GeWu-Lab/PSTP-Net}{https://github.com/GeWu-Lab/PSTP-Net}