EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
For audio language model researchers, EntangleCodec provides a single tokenizer that unifies understanding and generation with strong performance, demonstrating that representation quality can compensate for model scale.
EntangleCodec is a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations, achieving reconstruction quality competitive with specialized codecs, outperforming all codec-based baselines on audio understanding by up to +7.4% on MMAR, and enabling TTS and TTA generation. At 0.6B parameters, it surpasses 13B-parameter continuous-representation LLMs with 22× fewer parameters, and scaling to 8B sets new SOTA on MMAR.
Audio tokenizers serve as the discrete interface between continuous audio and Audio Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation. Reconstruction-oriented codecs preserve acoustic fidelity but lack rich semantics, while semantic-aware tokenizers typically rely on separate semantic and acoustic streams, introducing redundancy or misalignment. We propose \textbf{EntangleCodec}, a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations before quantization. By aligning audio with rich captions rather than ASR transcripts, EntangleCodec captures linguistic content, speaker identity, emotion, prosody, and acoustic scenes within a compact token stream. A flow-matching diffusion decoder further enables high-quality reconstruction across speech, music, and general audio. EntangleCodec achieves reconstruction quality competitive with specialized codecs, outperforms all codec-based baselines on audio understanding by up to \textbf{+7.4\%} on MMAR, and supports both TTS and TTA generation in a unified framework. Furthermore, EntangleCodec-based audio language models demonstrate strong scaling behavior: even at \textit{0.6B} parameters, the model surpasses specialized continuous-representation LLMs with over \textit{13B} parameters across three benchmarks using \textbf{22$\times$} fewer parameters; scaling to \textit{8B} further establishes new state-of-the-art results on MMAR, highlighting that representation quality is as critical as model scale in audio language modeling. Code and model weights are available at https://github.com/luckyerr/EntangleCodec.