A Study of Data Selection Strategies for Pre-training Self-Supervised Speech Models
For researchers and practitioners in speech processing, this work provides a practical data selection strategy that improves efficiency and performance, challenging the assumption that data diversity is crucial for SSL pre-training.
This study investigates data selection strategies for pre-training self-supervised speech models, finding that prioritizing the longest utterances yields superior ASR performance while using only half the original dataset and reducing pre-training time by 24%.
Self-supervised learning (SSL) has transformed speech processing, yet its reliance on massive pre-training datasets remains a bottleneck. While robustness is often attributed to scale and diversity, the role of the data distribution is less understood. We systematically examine how curated subsets of pre-training data influence Automatic Speech Recognition (ASR) performance. Surprisingly, optimizing for acoustic, speaker, or linguistic diversity yields no clear improvements over random sampling. Instead, we find that prioritizing the longest utterances achieves superior ASR results while using only half the original dataset, reducing pre-training time by 24% on a large corpora. These findings suggest that for pre-training speech SSL models, data length is a more critical factor than either data diversity or overall data quantity for performance and efficiency, offering a new perspective for data selection strategies in SSL speech processing.