LLM-based speaker diarization correction: A generalizable approach
This work addresses the issue of speaker diarization accuracy for users of automated speech recognition tools, but it is incremental as it builds on existing LLM methods for a known bottleneck.
The paper tackled the problem of low accuracy in speaker diarization by using fine-tuned large language models (LLMs) for correction as a post-processing step, reporting that these models can markedly improve diarization accuracy, with an ensemble model achieving better overall performance than ASR-specific models.
Speaker diarization is necessary for interpreting conversations transcribed using automated speech recognition (ASR) tools. Despite significant developments in diarization methods, diarization accuracy remains an issue. Here, we investigate the use of large language models (LLMs) for diarization correction as a post-processing step. LLMs were fine-tuned using the Fisher corpus, a large dataset of transcribed conversations. The ability of the models to improve diarization accuracy in a holdout dataset from the Fisher corpus as well as an independent dataset was measured. We report that fine-tuned LLMs can markedly improve diarization accuracy. However, model performance is constrained to transcripts produced using the same ASR tool as the transcripts used for fine-tuning, limiting generalizability. To address this constraint, an ensemble model was developed by combining weights from three separate models, each fine-tuned using transcripts from a different ASR tool. The ensemble model demonstrated better overall performance than each of the ASR-specific models, suggesting that a generalizable and ASR-agnostic approach may be achievable. We have made the weights of these models publicly available on HuggingFace at https://huggingface.co/bklynhlth.