Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers
This work addresses the scalability and efficiency limitations of separate speech models for researchers and practitioners, though it is incremental as it builds on existing LLM architectures.
The paper tackles the fragmentation of traditional speech systems by introducing General-Purpose Audio (GPA), a unified autoregressive transformer model that integrates text-to-speech, automatic speech recognition, and voice conversion into a single architecture, achieving competitive performance across these tasks with a lightweight 0.3B-parameter variant for edge deployment.
Traditional speech systems typically rely on separate, task-specific models for text-to-speech (TTS), automatic speech recognition (ASR), and voice conversion (VC), resulting in fragmented pipelines that limit scalability, efficiency, and cross-task generalization. In this paper, we present General-Purpose Audio (GPA), a unified audio foundation model that integrates multiple core speech tasks within a single large language model (LLM) architecture. GPA operates on a shared discrete audio token space and supports instruction-driven task induction, enabling a single autoregressive model to flexibly perform TTS, ASR, and VC without architectural modifications. This unified design combines a fully autoregressive formulation over discrete speech tokens, joint multi-task training across speech domains, and a scalable inference pipeline that achieves high concurrency and throughput. The resulting model family supports efficient multi-scale deployment, including a lightweight 0.3B-parameter variant optimized for edge and resource-constrained environments. Together, these design choices demonstrate that a unified autoregressive architecture can achieve competitive performance across diverse speech tasks while remaining viable for low-latency, practical deployment.