Can CLIP Help Sound Source Localization?
Sooyoung Park Arda Senocak Joon Son Chung
Large-scale pre-trained image-text models demonstrate remarkable versatility
across diverse tasks, benefiting from their robust representational
capabilities and effective multimodal alignment. We extend the application of
these models, specifically CLIP, to the domain of sound source localization.
Unlike conventional approaches, we employ the pre-trained CLIP model without
explicit text input, relying solely on the audio-visual correspondence. To this
end, we introduce a framework that translates audio signals into tokens
compatible with CLIP's text encoder, yielding audio-driven embeddings. By
directly using these embeddings, our method generates audio-grounded masks for
the provided audio, extracts audio-grounded image features from the highlighted
regions, and aligns them with the audio-driven embeddings using the
audio-visual correspondence objective. Our findings suggest that utilizing
pre-trained image-text models enable our model to generate more complete and
compact localization maps for the sounding objects. Extensive experiments show
that our method outperforms state-of-the-art approaches by a significant
margin.