Closing the Gap between Single-User and Multi-User VoiceFilter-Lite
This work addresses the limitation of speaker-conditioned speech models to support multiple users, which is crucial for smart home devices, representing an incremental improvement over previous multi-user models.
The paper tackled the performance degradation in multi-user VoiceFilter-Lite models compared to single-user versions by incorporating a dual learning rate schedule and feature-wise linear modulation (FiLM), successfully closing the gap on single-speaker evaluations and improving multi-speaker performance.
VoiceFilter-Lite is a speaker-conditioned voice separation model that plays a crucial role in improving speech recognition and speaker verification by suppressing overlapping speech from non-target speakers. However, one limitation of VoiceFilter-Lite, and other speaker-conditioned speech models in general, is that these models are usually limited to a single target speaker. This is undesirable as most smart home devices now support multiple enrolled users. In order to extend the benefits of personalization to multiple users, we previously developed an attention-based speaker selection mechanism and applied it to VoiceFilter-Lite. However, the original multi-user VoiceFilter-Lite model suffers from significant performance degradation compared with single-user models. In this paper, we devised a series of experiments to improve the multi-user VoiceFilter-Lite model. By incorporating a dual learning rate schedule and by using feature-wise linear modulation (FiLM) to condition the model with the attended speaker embedding, we successfully closed the performance gap between multi-user and single-user VoiceFilter-Lite models on single-speaker evaluations. At the same time, the new model can also be easily extended to support any number of users, and significantly outperforms our previously published model on multi-speaker evaluations.