Video-to-Music Recommendation using Temporal Alignment of Segments
Laure Prétet Gaël Richard Clément Souchier Geoffroy Peeters
We study cross-modal recommendation of music tracks to be used as soundtracks
for videos. This problem is known as the music supervision task. We build on a
self-supervised system that learns a content association between music and
video. In addition to the adequacy of content, adequacy of structure is crucial
in music supervision to obtain relevant recommendations. We propose a novel
approach to significantly improve the system's performance using
structure-aware recommendation. The core idea is to consider not only the full
audio-video clips, but rather shorter segments for training and inference. We
find that using semantic segments and ranking the tracks according to sequence
alignment costs significantly improves the results. We investigate the impact
of different ranking metrics and segmentation methods.