Sound Source Localization is All about Cross-Modal Alignment
Arda Senocak Hyeonggon Ryu Junsik Kim Tae-Hyun Oh Hanspeter Pfister Joon Son Chung
Humans can easily perceive the direction of sound sources in a visual scene,
termed sound source localization. Recent studies on learning-based sound source
localization have mainly explored the problem from a localization perspective.
However, prior arts and existing benchmarks do not account for a more important
aspect of the problem, cross-modal semantic understanding, which is essential
for genuine sound source localization. Cross-modal semantic understanding is
important in understanding semantically mismatched audio-visual events, e.g.,
silent objects, or off-screen sounds. To account for this, we propose a
cross-modal alignment task as a joint task with sound source localization to
better learn the interaction between audio and visual modalities. Thereby, we
achieve high localization performance with strong cross-modal semantic
understanding. Our method outperforms the state-of-the-art approaches in both
sound source localization and cross-modal retrieval. Our work suggests that
jointly tackling both tasks is necessary to conquer genuine sound source
localization.