A Real-Time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance
This work addresses the problem of automatic subtitling for live concerts or operas, though it is incremental as it builds on existing alignment methods with a new feature combination.
The paper tackles real-time lyrics alignment for classical vocal performances by combining chromagram and phonetic posteriorgram features, achieving a 15% improvement in alignment accuracy on the recast Schubert Winterreise Dataset.
The goal of real-time lyrics alignment is to take live singing audio as input and to pinpoint the exact position within given lyrics on the fly. The task can benefit real-world applications such as the automatic subtitling of live concerts or operas. However, designing a real-time model poses a great challenge due to the constraints of only using past input and operating within a minimal latency. Furthermore, due to the lack of datasets for real-time models for lyrics alignment, previous studies have mostly evaluated with private in-house datasets, resulting in a lack of standard evaluation methods. This paper presents a real-time lyrics alignment system for classical vocal performances with two contributions. First, we improve the lyrics alignment algorithm by finding an optimal combination of chromagram and phonetic posteriorgram (PPG) that capture melodic and phonetics features of the singing voice, respectively. Second, we recast the Schubert Winterreise Dataset (SWD) which contains multiple performance renditions of the same pieces as an evaluation set for the real-time lyrics alignment.