On Training Targets and Activation Functions for Deep Representation Learning in Text-Dependent Speaker Verification
This work addresses performance improvements in text-dependent speaker verification systems, particularly for short utterances, but is incremental as it focuses on optimizing existing components rather than introducing new paradigms.
The paper systematically studies the impact of training targets, activation functions, and loss functions on deep representation learning for text-dependent speaker verification, finding that GELU activation reduces error rates significantly compared to sigmoid, time-contrastive learning performs best among training targets, and cross entropy, joint-softmax, and focal loss outperform others, with score-level fusion further reducing errors.
Deep representation learning has gained significant momentum in advancing text-dependent speaker verification (TD-SV) systems. When designing deep neural networks (DNN) for extracting bottleneck features, key considerations include training targets, activation functions, and loss functions. In this paper, we systematically study the impact of these choices on the performance of TD-SV. For training targets, we consider speaker identity, time-contrastive learning (TCL) and auto-regressive prediction coding with the first being supervised and the last two being self-supervised. Furthermore, we study a range of loss functions when speaker identity is used as the training target. With regard to activation functions, we study the widely used sigmoid function, rectified linear unit (ReLU), and Gaussian error linear unit (GELU). We experimentally show that GELU is able to reduce the error rates of TD-SV significantly compared to sigmoid, irrespective of training target. Among the three training targets, TCL performs the best. Among the various loss functions, cross entropy, joint-softmax and focal loss functions outperform the others. Finally, score-level fusion of different systems is also able to reduce the error rates. Experiments are conducted on the RedDots 2016 challenge database for TD-SV using short utterances. For the speaker classifications, the well-known Gaussian mixture model-universal background model (GMM-UBM) and i-vector techniques are used.