HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models
For researchers building multimodal speech-text systems, this hybrid approach offers a practical improvement over discrete-only methods by balancing efficiency and quality.
The paper addresses performance degradation in speech language models due to information loss from discrete audio representations, proposing a hybrid discrete-continuous codec that improves speaker characteristic retention and reduces autoregressive steps.
Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilities into Large Language Models (LLMs). However, numerous studies report performance degradation on various downstream tasks due to information loss during discretization. To address this, we propose a novel approach combining temporally compressed discrete tokens with dimensionality-reduced continuous residuals. Our framework consists of a hybridized discrete-continuous focal modulation codec and a hybrid Transformer. This architecture performs autoregressive inference in the discrete domain, coupled with non-autoregressive prediction and continuous residual upsampling. Experimental results show that our approach significantly improves the retention of speaker characteristics compared to discrete-only methods, while simultaneously reducing the number of required autoregressive steps.