Do speech foundation models perceive speaker similarity as humans do?
For researchers developing speech models, this work highlights the gap between model-based and human-perceived speaker similarity, guiding improvements toward perceptually grounded embeddings.
This study compares speaker embeddings from over 40 speech foundation models with human subjective similarity ratings, finding that model distances do not fully align with human perception and identifying key configuration factors for better alignment.
This study presents a comparative analysis between the speaker embeddings of speech foundation models and human subjective perception of speaker similarity. Human listeners have the ability to judge speaker similarity on a continuous scale discerning how similar two voices are. In contrast, speech foundation models embed speaker characteristics into numerical representation. However, a question remains: does the numerical distance between speaker embeddings in these models truly align with the similarity perceived by humans? To address this, we conduct a comprehensive investigation using more than 40 models to compare model-derived distances with human-perceived similarity scores. Furthermore, we identify which factors in model configuration contribute most to a speaker embedding that mirrors human perception. Our findings provide insights for the development of more perceptually grounded speech foundation models.