Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization
This addresses a fundamental bottleneck in vector quantization for tokenization across modalities, with incremental improvements to existing methods.
The paper tackles the problem of premature discretization in vector quantization (VQ) for tokenization in multimodal models, proposing Progressive Quantization (ProVQ) to anneal from continuous to discrete spaces, resulting in improved reconstruction and generative performance on ImageNet benchmarks and setting a new state-of-the-art for protein structure tokenization.
Vector Quantization (VQ) has become the cornerstone of tokenization for many multimodal Large Language Models and diffusion synthesis. However, existing VQ paradigms suffer from a fundamental conflict: they enforce discretization before the encoder has captured the underlying data manifold. We term this phenomenon Premature Discretization. To resolve this, we propose Progressive Quantization (ProVQ), which incorporates the dynamics of quantization hardness as a fundamental yet previously overlooked axis in VQ training. By treating quantization as a curriculum that smoothly anneals from a continuous latent space to a discrete one, ProVQ effectively guides the codebook toward the well-expanded manifolds. Extensive experimental results demonstrate the broad effectiveness of ProVQ across diverse modalities. We report improved reconstruction and generative performance on the ImageNet-1K and ImageNet-100 benchmarks, highlighting the ProVQ's boost for generative modeling. Furthermore, ProVQ proves highly effective for modeling complex biological sequences, establishing a new performance ceiling for protein structure tokenization on the StrutTokenBench leaderboard.