Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokens
This addresses the challenge of transcribing meetings with unknown numbers of overlapping speakers, representing a strong specific gain in domain-specific applications.
The paper tackles the problem of multi-speaker speech recognition in overlapping speech scenarios by proposing a novel attention-based encoder-decoder method with speaker tokens and hypothesis clustering, achieving a 55% relative error reduction on clean data and 36% on noisy data in 3-mix environments compared to conventional methods.
In many real-world scenarios, such as meetings, multiple speakers are present with an unknown number of participants, and their utterances often overlap. We address these multi-speaker challenges by a novel attention-based encoder-decoder method augmented with special speaker class tokens obtained by speaker clustering. During inference, we select multiple recognition hypotheses conditioned on predicted speaker cluster tokens, and these hypotheses are merged by agglomerative hierarchical clustering (AHC) based on the normalized edit distance. The clustered hypotheses result in the multi-speaker transcriptions with the appropriate number of speakers determined by AHC. Our experiments on the LibriMix dataset demonstrate that our proposed method was particularly effective in complex 3-mix environments, achieving a 55% relative error reduction on clean data and a 36% relative error reduction on noisy data compared with conventional serialized output training.