OpusLM: A Family of Open Unified Speech Language Models
This provides open and transparent models for speech-language research, addressing a need for accessible tools in the AI community, though it is incremental as it builds on existing text language models and training strategies.
The paper tackles the problem of developing open foundational speech language models (SpeechLMs) by introducing OpusLMs, a family of models up to 7B parameters trained on 213K hours of speech-text pairs and 292B text tokens, achieving comparable or superior performance to existing SpeechLMs in speech recognition, synthesis, and text tasks.
This paper presents Open Unified Speech Language Models (OpusLMs), a family of open foundational speech language models (SpeechLMs) up to 7B. Initialized from decoder-only text language models, the OpusLMs are continuously pre-trained on 213K hours of speech-text pairs and 292B text-only tokens. We demonstrate our OpusLMs achieve comparable (or even superior) performance with existing SpeechLMs in speech recognition, speech synthesis, and text-only capabilities. Technically, this paper articulates our SpeechLM designs on tokenization, multi-stream language models, and multi-stage training strategies. We experimentally demonstrate the importance of model size scaling and the effect of annealing data selection. The OpusLMs are all built from publicly available materials and are fully transparent models. We release our code, data, checkpoints, and training logs to facilitate open SpeechLM research