Video-to-Music Recommendation using Temporal Alignment of Segments
This addresses the music supervision task for video creators, offering an incremental improvement over existing methods.
The paper tackles the problem of recommending music tracks as soundtracks for videos by improving a self-supervised system with structure-aware recommendation, using semantic segments and sequence alignment to achieve significant performance gains.
We study cross-modal recommendation of music tracks to be used as soundtracks for videos. This problem is known as the music supervision task. We build on a self-supervised system that learns a content association between music and video. In addition to the adequacy of content, adequacy of structure is crucial in music supervision to obtain relevant recommendations. We propose a novel approach to significantly improve the system's performance using structure-aware recommendation. The core idea is to consider not only the full audio-video clips, but rather shorter segments for training and inference. We find that using semantic segments and ranking the tracks according to sequence alignment costs significantly improves the results. We investigate the impact of different ranking metrics and segmentation methods.