BabyLM's First Words: Word Segmentation as a Phonological Probing Task
This work addresses the problem of limited phonological benchmarks for researchers in linguistics and AI, though it is incremental as it builds on existing computational models of word segmentation.
The paper tackles the difficulty of phonological analysis in large language models by using word segmentation as a probing task across 31 languages, showing that phoneme-based models trained on child-directed speech can implicitly track word boundaries and corroborate statistical learning theories.
Language models provide a key framework for studying linguistic theories based on prediction, but phonological analysis using large language models (LLMs) is difficult; there are few phonological benchmarks beyond English and the standard input representation used in LLMs (subwords of graphemes) is not suitable for analyzing the representation of phonemes. In this work, we demonstrate how word segmentation can be used as a phonological probing task, allowing us to study the representations learned by phoneme-based language models trained on child-directed speech across 31 languages. Following computational models of word segmentation, we present unsupervised methods for extracting word boundaries from a trained model using the observation that prediction-error peaks at the start of words. We also use linear probes to identify that these models implicitly track word boundaries, even when they do not appear in training. This cross-lingual work corroborates statistical learning theories of acquisition and empirically motivates new methods for training subword tokenizers.