Self-Supervised Speech Representations are More Phonetic than Semantic
This work addresses the problem of understanding linguistic encoding in speech models for researchers and practitioners, but it is incremental as it builds on prior analyses.
The study analyzed self-supervised speech models (S3Ms) and found that their word representations show significantly more phonetic than semantic similarity, as measured using near homophone and synonym pairs. It also revealed that simple baselines using word identity outperform S3M-based models on intent classification datasets, indicating these datasets may not adequately measure semantic abilities.
Self-supervised speech models (S3Ms) have become an effective backbone for speech applications. Various analyses suggest that S3Ms encode linguistic properties. In this work, we seek a more fine-grained analysis of the word-level linguistic properties encoded in S3Ms. Specifically, we curate a novel dataset of near homophone (phonetically similar) and synonym (semantically similar) word pairs and measure the similarities between S3M word representation pairs. Our study reveals that S3M representations consistently and significantly exhibit more phonetic than semantic similarity. Further, we question whether widely used intent classification datasets such as Fluent Speech Commands and Snips Smartlights are adequate for measuring semantic abilities. Our simple baseline, using only the word identity, surpasses S3M-based models. This corroborates our findings and suggests that high scores on these datasets do not necessarily guarantee the presence of semantic content.