An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training
This work addresses the optimization of speech modeling for researchers in speech processing, but it is incremental as it builds on existing pre-trained language models.
The paper investigates how discrete unit representations affect speech language model pre-training, finding that optimal discretization strategies depend on model capacity and that domain matching in clustering data selection is crucial for robustness.
This paper investigates discrete unit representations in Speech Language Models (SLMs), focusing on optimizing speech modeling during continual pre-training. In this paper, we systematically examine how model architecture, data representation, and training robustness influence the pre-training stage in which we adapt existing pre-trained language models to the speech modality. Our experiments highlight the role of speech encoders and clustering granularity across different model scales, showing how optimal discretization strategies vary with model capacity. By examining cluster distribution and phonemic alignments, we investigate the effective use of discrete vocabulary, uncovering both linguistic and paralinguistic patterns. Additionally, we explore the impact of clustering data selection on model robustness, highlighting the importance of domain matching between discretization training and target applications.