Bridging High-Quality Audio and Video via Language for Sound Effects
  Retrieval from Visual Queries
Julia Wilkins Justin Salamon Magdalena Fuentes Juan Pablo Bello Oriol Nieto
Finding the right sound effects (SFX) to match moments in a video is a
difficult and time-consuming task, and relies heavily on the quality and
completeness of text metadata. Retrieving high-quality (HQ) SFX using a video
frame directly as the query is an attractive alternative, removing the reliance
on text metadata and providing a low barrier to entry for non-experts. Due to
the lack of HQ audio-visual training data, previous work on audio-visual
retrieval relies on YouTube (in-the-wild) videos of varied quality for
training, where the audio is often noisy and the video of amateur quality. As
such it is unclear whether these systems would generalize to the task of
matching HQ audio to production-quality video. To address this, we propose a
multimodal framework for recommending HQ SFX given a video frame by (1)
leveraging large language models and foundational vision-language models to
bridge HQ audio and video to create audio-visual pairs, resulting in a highly
scalable automatic audio-visual data curation pipeline; and (2) using
pre-trained audio and visual encoders to train a contrastive learning-based
retrieval system. We show that our system, trained using our automatic data
curation pipeline, significantly outperforms baselines trained on in-the-wild
data on the task of HQ SFX retrieval for video. Furthermore, while the
baselines fail to generalize to this task, our system generalizes well from
clean to in-the-wild data, outperforming the baselines on a dataset of YouTube
videos despite only being trained on the HQ audio-visual pairs. A user study
confirms that people prefer SFX retrieved by our system over the baseline 67%
of the time both for HQ and in-the-wild data. Finally, we present ablations to
determine the impact of model and data pipeline design choices on downstream
retrieval performance. Please visit our project website to listen to and view
our SFX retrieval results.