Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages
This work addresses the need for accurate and simple lyrics alignment tools for music processing applications, offering a novel approach that improves over existing methods.
The paper tackles the problem of aligning audio with lyrics across multiple languages by introducing a contrastive learning-based system, which achieves an average absolute error below 0.2 seconds on the standard Jamendo dataset and demonstrates robustness to other languages even when trained only on English data.
Lyrics alignment gained considerable attention in recent years. State-of-the-art systems either re-use established speech recognition toolkits, or design end-to-end solutions involving a Connectionist Temporal Classification (CTC) loss. However, both approaches suffer from specific weaknesses: toolkits are known for their complexity, and CTC systems use a loss designed for transcription which can limit alignment accuracy. In this paper, we use instead a contrastive learning procedure that derives cross-modal embeddings linking the audio and text domains. This way, we obtain a novel system that is simple to train end-to-end, can make use of weakly annotated training data, jointly learns a powerful text model, and is tailored to alignment. The system is not only the first to yield an average absolute error below 0.2 seconds on the standard Jamendo dataset but it is also robust to other languages, even when trained on English data only. Finally, we release word-level alignments for the JamendoLyrics Multi-Lang dataset.