HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Model
This work addresses the problem of enhancing semantic representation in self-supervised speech models for tasks like speech recognition, though it is incremental as it builds on existing HuBERT with a novel auxiliary task.
The paper tackles the limitation of HuBERT's masked prediction task in capturing global semantic information by proposing HuBERTopic, which enriches semantic representation through a topic model and auxiliary classification. The method achieves comparable or better performance than the baseline in most tasks, including automatic speech recognition and five out of eight SUPERB tasks.
Recently, the usefulness of self-supervised representation learning (SSRL) methods has been confirmed in various downstream tasks. Many of these models, as exemplified by HuBERT and WavLM, use pseudo-labels generated from spectral features or the model's own representation features. From previous studies, it is known that the pseudo-labels contain semantic information. However, the masked prediction task, the learning criterion of HuBERT, focuses on local contextual information and may not make effective use of global semantic information such as speaker, theme of speech, and so on. In this paper, we propose a new approach to enrich the semantic representation of HuBERT. We apply topic model to pseudo-labels to generate a topic label for each utterance. An auxiliary topic classification task is added to HuBERT by using topic labels as teachers. This allows additional global semantic information to be incorporated in an unsupervised manner. Experimental results demonstrate that our method achieves comparable or better performance than the baseline in most tasks, including automatic speech recognition and five out of the eight SUPERB tasks. Moreover, we find that topic labels include various information about utterance, such as gender, speaker, and its theme. This highlights the effectiveness of our approach in capturing multifaceted semantic nuances.