Synchronising speech segments with musical beats in Mandarin and English singing
This addresses the issue of off-beat synthesized singing for voice synthesis applications, but it is incremental as it analyzes existing data rather than proposing a new method.
The study tackled the problem of synthesizing singing voice from speech data by investigating the temporal relationship between speech segments and musical beats in Mandarin and English singing. The results showed that beat presence depends more on segment duration than sonority, with cross-linguistic variations observed.
Generating synthesised singing voice with models trained on speech data has many advantages due to the models' flexibility and controllability. However, since the information about the temporal relationship between segments and beats are lacking in speech training data, the synthesised singing may sound off-beat at times. Therefore, the availability of the information on the temporal relationship between speech segments and music beats is crucial. The current study investigated the segment-beat synchronisation in singing data, with hypotheses formed based on the linguistics theories of P-centre and sonority hierarchy. A Mandarin corpus and an English corpus of professional singing data were manually annotated and analysed. The results showed that the presence of musical beats was more dependent on segment duration than sonority. However, the sonority hierarchy and the P-centre theory were highly related to the location of beats. Mandarin and English demonstrated cross-linguistic variations despite exhibiting common patterns.