Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformers
This work addresses a gap in knowledge for researchers and practitioners in speech processing, but it is incremental as it builds on existing models without introducing new paradigms.
The paper tackled the problem of understanding how self-supervised speech Transformers encode speaker information by identifying neurons in feed-forward layers correlated with speaker data, and found that protecting these neurons during pruning significantly preserves performance on speaker-related tasks.
In recent years, the impact of self-supervised speech Transformers has extended to speaker-related applications. However, little research has explored how these models encode speaker information. In this work, we address this gap by identifying neurons in the feed-forward layers that are correlated with speaker information. Specifically, we analyze neurons associated with k-means clusters of self-supervised features and i-vectors. Our analysis reveals that these clusters correspond to broad phonetic and gender classes, making them suitable for identifying neurons that represent speakers. By protecting these neurons during pruning, we can significantly preserve performance on speaker-related task, demonstrating their crucial role in encoding speaker information.