CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding
For speech processing pipelines, CleanCodec provides a more efficient and robust tokenization method that improves downstream task performance and speed.
CleanCodec reframes audio tokenization as a selective information bottleneck, achieving state-of-the-art tokenization efficiency at 12.5 tokens per second while outperforming existing codecs in speaker similarity and speech intelligibility, with up to 17x faster inference in downstream tasks.
Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content. We reframe audio tokenization as a selective information bottleneck problem and propose CleanCodec, a denoising audio codec which learns to encode only perceptually important features and discard imperceptible information. At just 12.5 tokens per second, CleanCodec achieves state-of-the-art tokenization efficiency, substantially outperforming existing codecs in speaker similarity and speech intelligibility. Evaluations on downstream text-to-speech and voice conversion tasks further demonstrate improved performance and up to 17x faster inference, highlighting significant efficiency gains.