NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference
This addresses the need for low-latency and efficient speech LLM training and inference, representing a strong specific gain in audio processing.
The paper tackled the problem of slow training and inference in speech LLMs due to high frame-rate audio codecs by introducing NanoCodec, which achieves high-quality compression at 12.5 FPS and outperforms related works across bitrate ranges.
Large Language Models (LLMs) have significantly advanced audio processing by leveraging audio codecs to discretize audio into tokens, enabling the application of language modeling techniques to speech data. However, existing audio codecs often operate at high frame rates, leading to slow training and inference, particularly for autoregressive models. To address this, there is growing interest in low frame-rate audio codecs, which reduce the number of autoregressive steps required to generate one second of audio. In this paper, we conduct ablation studies to examine the impact of frame rate, bitrate, and causality on codec reconstruction quality. Based on our findings, we introduce NanoCodec, a state-of-the-art audio codec that achieves high-quality compression at just 12.5 frames per second (FPS). NanoCodec outperforms related works across various bitrate ranges, establishing a new benchmark for low-latency and efficient Speech LLM training and inference.