FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation
This addresses the need for low-latency, high-quality speech coding in real-time applications like communication systems, though it is incremental with targeted architectural improvements.
The paper tackled the problem of non-streamable neural audio codecs for real-time applications by introducing FocalCodec-Stream, a hybrid codec that compresses speech at 0.55-0.80 kbps with 80 ms latency and outperforms existing streamable codecs in reconstruction quality and downstream task performance.
Neural audio codecs are a fundamental component of modern generative audio pipelines. Although recent codecs achieve strong low-bitrate reconstruction and provide powerful representations for downstream tasks, most are non-streamable, limiting their use in real-time applications. We present FocalCodec-Stream, a hybrid codec based on focal modulation that compresses speech into a single binary codebook at 0.55 - 0.80 kbps with a theoretical latency of 80 ms. Our approach combines multi-stage causal distillation of WavLM with targeted architectural improvements, including a lightweight refiner module that enhances quality under latency constraints. Experiments show that FocalCodec-Stream outperforms existing streamable codecs at comparable bitrates, while preserving both semantic and acoustic information. The result is a favorable trade-off between reconstruction quality, downstream task performance, latency, and efficiency. Code and checkpoints will be released at https://github.com/lucadellalib/focalcodec.