Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition
This work addresses multi-speaker ASR for applications like transcription in noisy environments, but it is incremental as it builds on existing SOT and hybrid loss methods.
The paper tackles the challenge of training multi-speaker automatic speech recognition using serialized output training by proposing overlapped encoding separation and serialized speech information guidance to leverage CTC and attention hybrid losses, resulting in improved encoder representation and performance on the LibriMix dataset.
Serialized output training (SOT) attracts increasing attention due to its convenience and flexibility for multi-speaker automatic speech recognition (ASR). However, it is not easy to train with attention loss only. In this paper, we propose the overlapped encoding separation (EncSep) to fully utilize the benefits of the connectionist temporal classification (CTC) and attention hybrid loss. This additional separator is inserted after the encoder to extract the multi-speaker information with CTC losses. Furthermore, we propose the serialized speech information guidance SOT (GEncSep) to further utilize the separated encodings. The separated streams are concatenated to provide single-speaker information to guide attention during decoding. The experimental results on LibriMix show that the single-speaker encoding can be separated from the overlapped encoding. The CTC loss helps to improve the encoder representation under complex scenarios. GEncSep further improved performance.