Self-supervised speech representation learning for keyword-spotting with light-weight transformers
This work addresses keyword-spotting for compute-constrained applications, offering an incremental improvement over existing methods.
The paper tackled keyword-spotting on resource-constrained devices by applying self-supervised speech representation learning to light-weight transformers, achieving a 1.2% accuracy improvement on a public dataset and up to 23.7% relative false accept improvement on an in-house dataset.
Self-supervised speech representation learning (S3RL) is revolutionizing the way we leverage the ever-growing availability of data. While S3RL related studies typically use large models, we employ light-weight networks to comply with tight memory of compute-constrained devices. We demonstrate the effectiveness of S3RL on a keyword-spotting (KS) problem by using transformers with 330k parameters and propose a mechanism to enhance utterance-wise distinction, which proves crucial for improving performance on classification tasks. On the Google speech commands v2 dataset, the proposed method applied to the Auto-Regressive Predictive Coding S3RL led to a 1.2% accuracy improvement compared to training from scratch. On an in-house KS dataset with four different keywords, it provided 6% to 23.7% relative false accept improvement at fixed false reject rate. We argue this demonstrates the applicability of S3RL approaches to light-weight models for KS and confirms S3RL is a powerful alternative to traditional supervised learning for resource-constrained applications.