On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models
This work provides insights for optimizing bitrate in GSLM, potentially reducing computational costs for speech generation tasks, but the findings are incremental as they primarily confirm that lower bitrates suffice without introducing new methods.
The paper investigates how segmentation width and cluster size affect speech resynthesis and continuation in Generative Spoken Language Models, finding that lower bitrate settings than the baseline can produce intelligible and natural speech, and that speech continuation quality remains stable at lower bitrates, suggesting the conventional GSLM setting may be redundant.
Generative Spoken Language Modeling (GSLM) enables text-free speech modeling by training language models (LMs) using discrete speech representations instead of textual transcription. In this paper, we investigate the performance of GSLM on speech synthesis and continuation using discrete speech representations with varying bitrates. We segment speech representations with fixed widths and train K-means models in multiple cluster sizes, resulting in various bitrate settings. We demonstrate that intelligible and natural speech can be synthesized at lower bitrate settings than the baseline. Furthermore, speech continuation quality remains stable at lower bitrates across multiple metrics, suggesting that the conventional GSLM setting may be redundant for effective speech generation. Although LLM-based metrics show higher correlation with human subjective score than conventional metrics, it remains low, highlighting the need for more stable automatic evaluation methods.