Auxiliary Loss of Transformer with Residual Connection for End-to-End Speaker Diarization
This work addresses speaker diarization for handling overlapped speech in applications like call analysis, but it is incremental as it builds on existing EEND methods.
The paper tackles the problem of improving end-to-end neural diarization (EEND) for speaker diarization by addressing the issue of dominant supervision in the last encoder block, proposing a residual auxiliary learning architecture (RX-EEND) that applies auxiliary loss to each encoder block and adds residual connections. The result shows relative reductions in diarization error rate (DER) by 50.3% and 21.0% on simulated and CALLHOME datasets compared to baseline, with further reductions of 8.1% on CALLHOME from residual connections.
End-to-end neural diarization (EEND) with self-attention directly predicts speaker labels from inputs and enables the handling of overlapped speech. Although the EEND outperforms clustering-based speaker diarization (SD), it cannot be further improved by simply increasing the number of encoder blocks because the last encoder block is dominantly supervised compared with lower blocks. This paper proposes a new residual auxiliary EEND (RX-EEND) learning architecture for transformers to enforce the lower encoder blocks to learn more accurately. The auxiliary loss is applied to the output of each encoder block, including the last encoder block. The effect of auxiliary loss on the learning of the encoder blocks can be further increased by adding a residual connection between the encoder blocks of the EEND. Performance evaluation and ablation study reveal that the auxiliary loss in the proposed RX-EEND provides relative reductions in the diarization error rate (DER) by 50.3% and 21.0% on the simulated and CALLHOME (CH) datasets, respectively, compared with self-attentive EEND (SA-EEND). Furthermore, the residual connection used in RX-EEND further relatively reduces the DER by 8.1% for CH dataset.