Distilling Multi-Level X-vector Knowledge for Small-footprint Speaker Verification
This work addresses the deployment challenge of deep speaker models for speaker verification, offering a domain-specific solution that is incremental in nature.
The paper tackled the problem of reducing model size and computation for speaker verification in resource-constrained environments by using knowledge distillation to combine multi-level embeddings from an x-vector teacher network, achieving a 75% reduction in model size while maintaining comparable performance.
Even though deep speaker models have demonstrated impressive accuracy in speaker verification tasks, this often comes at the expense of increased model size and computation time, presenting challenges for deployment in resource-constrained environments. Our research focuses on addressing this limitation through the development of small footprint deep speaker embedding extraction using knowledge distillation. While previous work in this domain has concentrated on speaker embedding extraction at the utterance level, our approach involves amalgamating embeddings from different levels of the x-vector model (teacher network) to train a compact student network. The results highlight the significance of frame-level information, with the student models exhibiting a remarkable size reduction of 85%-91% compared to their teacher counterparts, depending on the size of the teacher embeddings. Notably, by concatenating teacher embeddings, we achieve student networks that maintain comparable performance to the teacher while enjoying a substantial 75% reduction in model size. These findings and insights extend to other x-vector variants, underscoring the broad applicability of our approach.