Inference-Time Scaling for Visual AutoRegressive modeling by Searching Representative Samples
This work addresses a critical bottleneck in visual autoregressive modeling for researchers and practitioners, offering a novel method to enhance generative quality, though it is incremental as it adapts existing scaling techniques to a new domain.
The authors tackled the challenge of applying inference-time scaling to vector-quantized visual autoregressive models by introducing VAR-Scaling, which maps discrete latent spaces to quasi-continuous ones using kernel density estimation and employs a hybrid sampling strategy, resulting in significant improvements in inference process quality as demonstrated in class-conditional and text-to-image evaluations.
While inference-time scaling has significantly enhanced generative quality in large language and diffusion models, its application to vector-quantized (VQ) visual autoregressive modeling (VAR) remains unexplored. We introduce VAR-Scaling, the first general framework for inference-time scaling in VAR, addressing the critical challenge of discrete latent spaces that prohibit continuous path search. We find that VAR scales exhibit two distinct pattern types: general patterns and specific patterns, where later-stage specific patterns conditionally optimize early-stage general patterns. To overcome the discrete latent space barrier in VQ models, we map sampling spaces to quasi-continuous feature spaces via kernel density estimation (KDE), where high-density samples approximate stable, high-quality solutions. This transformation enables effective navigation of sampling distributions. We propose a density-adaptive hybrid sampling strategy: Top-k sampling focuses on high-density regions to preserve quality near distribution modes, while Random-k sampling explores low-density areas to maintain diversity and prevent premature convergence. Consequently, VAR-Scaling optimizes sample fidelity at critical scales to enhance output quality. Experiments in class-conditional and text-to-image evaluations demonstrate significant improvements in inference process. The code is available at https://github.com/WD7ang/VAR-Scaling.