General-Purpose Speech Representation Learning through a Self-Supervised Multi-Granularity Framework
This work aims to improve general-purpose speech representations for the broader speech processing community by proposing a multi-granularity self-supervised learning framework.
This paper introduces MGF, a self-supervised multi-granularity framework for general-purpose speech representation learning. It combines generative learning for fine-grained information and discriminative learning for coarse-grained semantic information, showing improved performance across various downstream speech tasks.
This paper presents a self-supervised learning framework, named MGF, for general-purpose speech representation learning. In the design of MGF, speech hierarchy is taken into consideration. Specifically, we propose to use generative learning approaches to capture fine-grained information at small time scales and use discriminative learning approaches to distill coarse-grained or semantic information at large time scales. For phoneme-scale learning, we borrow idea from the masked language model but tailor it for the continuous speech signal by replacing classification loss with a contrastive loss. We corroborate our design by evaluating MGF representation on various downstream tasks, including phoneme classification, speaker classification, speech recognition, and emotion classification. Experiments verify that training at different time scales needs different training targets and loss functions, which in general complement each other and lead to a better performance.