RandAR: Decoder-only Autoregressive Visual Generation in Random Orders
This addresses the efficiency and flexibility bottlenecks in visual generation for researchers and practitioners, though it is incremental as it builds on existing decoder-only transformer frameworks.
The paper tackles the limitation of predefined generation orders in decoder-only autoregressive visual models by introducing RandAR, which generates images in arbitrary token orders and achieves comparable performance to fixed-order models while enabling new capabilities like zero-shot inpainting and 2.5x faster inference.
We introduce RandAR, a decoder-only visual autoregressive (AR) model capable of generating images in arbitrary token orders. Unlike previous decoder-only AR models that rely on a predefined generation order, RandAR removes this inductive bias, unlocking new capabilities in decoder-only generation. Our essential design enables random order by inserting a "position instruction token" before each image token to be predicted, representing the spatial location of the next image token. Trained on randomly permuted token sequences -- a more challenging task than fixed-order generation, RandAR achieves comparable performance to its conventional raster-order counterpart. More importantly, decoder-only transformers trained from random orders acquire new capabilities. For the efficiency bottleneck of AR models, RandAR adopts parallel decoding with KV-Cache at inference time, enjoying 2.5x acceleration without sacrificing generation quality. Additionally, RandAR supports inpainting, outpainting and resolution extrapolation in a zero-shot manner. We hope RandAR inspires new directions for decoder-only visual generation models and broadens their applications across diverse scenarios. Our project page is at https://rand-ar.github.io/.