Fine-Tuned Self-Supervised Speech Representations for Language Diarization in Multilingual Code-Switched Speech
This addresses the costly and time-consuming annotation process for multilingual code-switched speech, enabling parallel assignment to language experts, though it is incremental as it builds on existing self-supervised methods.
The paper tackles the problem of annotating multilingual code-switched speech by developing a continuous language diarizer using fine-tuned WavLM representations, resulting in substantial diarization error rate improvements for language families, groups, and individual languages in a South African corpus.
Annotating a multilingual code-switched corpus is a painstaking process requiring specialist linguistic expertise. This is partly due to the large number of language combinations that may appear within and across utterances, which might require several annotators with different linguistic expertise to consider an utterance sequentially. This is time-consuming and costly. It would be useful if the spoken languages in an utterance and the boundaries thereof were known before annotation commences, to allow segments to be assigned to the relevant language experts in parallel. To address this, we investigate the development of a continuous multilingual language diarizer using fine-tuned speech representations extracted from a large pre-trained self-supervised architecture (WavLM). We experiment with a code-switched corpus consisting of five South African languages (isiZulu, isiXhosa, Setswana, Sesotho and English) and show substantial diarization error rate improvements for language families, language groups, and individual languages over baseline systems.