Visual Autoregressive Models Beat Diffusion Models on Inference Time Scaling
This work addresses the problem of inefficient inference-time optimization in image generation for AI researchers and practitioners, offering a novel architectural approach that is not incremental but shifts focus from scale to model design.
The paper tackled the challenge of applying inference-time scaling to image generation by showing that visual autoregressive models, due to their discrete token space, enable effective search strategies like beam search, resulting in a 2B parameter model outperforming a 12B parameter diffusion model across benchmarks.
While inference-time scaling through search has revolutionized Large Language Models, translating these gains to image generation has proven difficult. Recent attempts to apply search strategies to continuous diffusion models show limited benefits, with simple random sampling often performing best. We demonstrate that the discrete, sequential nature of visual autoregressive models enables effective search for image generation. We show that beam search substantially improves text-to-image generation, enabling a 2B parameter autoregressive model to outperform a 12B parameter diffusion model across benchmarks. Systematic ablations show that this advantage comes from the discrete token space, which allows early pruning and computational reuse, and our verifier analysis highlights trade-offs between speed and reasoning capability. These findings suggest that model architecture, not just scale, is critical for inference-time optimization in visual generation.