F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
For audio AI researchers, this work addresses the mismatch between reconstruction-focused autoencoders and semantic-focused encoders, enabling a single tokenizer for both tasks.
F3-Tokenizer adapts continuous audio autoencoder latents to support both understanding and generation by introducing a noise-regularized bottleneck and a latent-side representation encoder, achieving competitive reconstruction and understanding performance while enabling autoregressive generation.
Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable. This mismatch complicates a single audio tokenizer that must support both understanding and generation. We adapt continuous autoencoder latents to this setting with two components: a noise-regularized autoencoder bottleneck and a latent-side representation encoder. The bottleneck uses channel normalization and stochastic perturbation instead of KL-based variational training, yielding scale-controlled continuous latents for reconstruction and autoregressive generation. The representation encoder is trained on frozen autoencoder latents with RQ-MTP and frozen-LLM supervision. The resulting tokenizer provides high-dimensional representations for understanding while preserving normalized continuous latents as generation targets