LSRS: Latent Scale Rejection Sampling for Visual Autoregressive Modeling
This work addresses a specific bottleneck in VAR models for image generation, offering an incremental improvement in generation quality with efficient test-time scaling.
The paper tackles the problem of structural errors in Visual Autoregressive (VAR) modeling for image generation by proposing Latent Scale Rejection Sampling (LSRS), which refines token maps during inference to improve quality with minimal computational overhead, reducing FID scores from 1.95 to 1.78 with only a 1% increase in inference time.
Visual Autoregressive (VAR) modeling approach for image generation proposes autoregressive processing across hierarchical scales, decoding multiple tokens per scale in parallel. This method achieves high-quality generation while accelerating synthesis. However, parallel token sampling within a scale may lead to structural errors, resulting in suboptimal generated images. To mitigate this, we propose Latent Scale Rejection Sampling (LSRS), a method that progressively refines token maps in the latent scale during inference to enhance VAR models. Our method uses a lightweight scoring model to evaluate multiple candidate token maps sampled at each scale, selecting the high-quality map to guide subsequent scale generation. By prioritizing early scales critical for structural coherence, LSRS effectively mitigates autoregressive error accumulation while maintaining computational efficiency. Experiments demonstrate that LSRS significantly improves VAR's generation quality with minimal additional computational overhead. For the VAR-d30 model, LSRS increases the inference time by merely 1% while reducing its FID score from 1.95 to 1.78. When the inference time is increased by 15%, the FID score can be further reduced to 1.66. LSRS offers an efficient test-time scaling solution for enhancing VAR-based generation.