DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generation
This addresses the problem of high computational and memory demands in autoregressive visual generation for researchers and practitioners, offering an incremental improvement with specific gains.
The paper tackles the computational inefficiency of fixed-length tokenization in autoregressive image generation by introducing DPAR, a model that dynamically aggregates tokens into variable-sized patches based on information content, reducing token counts by up to 2.06x and improving FID by up to 27.1%.
Decoder-only autoregressive image generation typically relies on fixed-length tokenization schemes whose token counts grow quadratically with resolution, substantially increasing the computational and memory demands of attention. We present DPAR, a novel decoder-only autoregressive model that dynamically aggregates image tokens into a variable number of patches for efficient image generation. Our work is the first to demonstrate that next-token prediction entropy from a lightweight and unsupervised autoregressive model provides a reliable criterion for merging tokens into larger patches based on information content. DPAR makes minimal modifications to the standard decoder architecture, ensuring compatibility with multimodal generation frameworks and allocating more compute to generation of high-information image regions. Further, we demonstrate that training with dynamically sized patches yields representations that are robust to patch boundaries, allowing DPAR to scale to larger patch sizes at inference. DPAR reduces token count by 1.81x and 2.06x on Imagenet 256 and 384 generation resolution respectively, leading to a reduction of up to 40% FLOPs in training costs. Further, our method exhibits faster convergence and improves FID by up to 27.1% relative to baseline models.