Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network
This addresses the need for more accurate speaker-attributed transcription in applications like meeting analysis, though it is incremental as it builds on existing end-to-end neural diarization methods.
The paper tackles the problem of jointly performing automatic speech recognition and speaker diarization at the word level, proposing WEEND, which outperforms turn-based baselines in 2-speaker scenarios and generalizes to 5-minute audio.
While standard speaker diarization attempts to answer the question "who spoken when", most of relevant applications in reality are more interested in determining "who spoken what". Whether it is the conventional modularized approach or the more recent end-to-end neural diarization (EEND), an additional automatic speech recognition (ASR) model and an orchestration algorithm are required to associate the speaker labels with recognized words. In this paper, we propose Word-level End-to-End Neural Diarization (WEEND) with auxiliary network, a multi-task learning algorithm that performs end-to-end ASR and speaker diarization in the same neural architecture. That is, while speech is being recognized, speaker labels are predicted simultaneously for each recognized word. Experimental results demonstrate that WEEND outperforms the turn-based diarization baseline system on all 2-speaker short-form scenarios and has the capability to generalize to audio lengths of 5 minutes. Although 3+speaker conversations are harder, we find that with enough in-domain training data, WEEND has the potential to deliver high quality diarized text.