Cross-Lingual Interleaving for Speech Language Models
This work addresses the challenge of building multilingual SLMs for languages with limited written resources, though it is incremental as it builds on existing SLM methods with a novel training approach.
The paper tackles the problem of English-centric progress in Spoken Language Models (SLMs) by introducing a cross-lingual interleaving method that mixes speech tokens across languages without textual supervision, resulting in improved monolingual semantic accuracy, robust cross-lingual continuation, and strengthened cross-lingual hidden-state alignment for 360M and 1B SLMs under matched training-token budgets.
Spoken Language Models (SLMs) aim to learn linguistic competence directly from speech using discrete units, widening access to Natural Language Processing (NLP) technologies for languages with limited written resources. However, progress has been largely English-centric due to scarce spoken evaluation benchmarks and training data, making cross-lingual learning difficult. We present a cross-lingual interleaving method that mixes speech tokens across languages without textual supervision. We also release an EN-FR training dataset, TinyStories (~42k hours), together with EN-FR spoken StoryCloze and TopicCloze benchmarks for cross-lingual semantic evaluation, both synthetically generated using GPT-4. On 360M and 1B SLMs under matched training-token budgets, interleaving improves monolingual semantic accuracy, enables robust cross-lingual continuation, and strengthens cross-lingual hidden-state alignment. Taken together, these results indicate that cross-lingual interleaving is a simple, scalable route to building multilingual SLMs that understand and converse across languages. All resources will be made open-source to support reproducibility.