Boosting Resolution Generalization of Diffusion Transformers with Randomized Positional Encodings
This addresses a key challenge in image generation for AI researchers and practitioners by enabling efficient high-resolution image production without high-resolution training, though it is incremental as it builds on existing Diffusion Transformer methods.
The paper tackles the problem of resolution generalization in Diffusion Transformers for image generation by proposing a novel two-dimensional randomized positional encodings (RPE-2D) framework, which achieves state-of-the-art performance on ImageNet, outperforming existing methods when trained at 256x256 and inferred at higher resolutions like 384x384 and 512x512, and scaling up to 1024x1024.
Resolution generalization in image generation tasks enables the production of higher-resolution images with lower training resolution overhead. However, a significant challenge in resolution generalization, particularly in the widely used Diffusion Transformers, lies in the mismatch between the positional encodings encountered during testing and those used during training. While existing methods have employed techniques such as interpolation, extrapolation, or their combinations, none have fully resolved this issue. In this paper, we propose a novel two-dimensional randomized positional encodings (RPE-2D) framework that focuses on learning positional order of image patches instead of the specific distances between them, enabling seamless high- and low-resolution image generation without requiring high- and low-resolution image training. Specifically, RPE-2D independently selects positions over a broader range along both the horizontal and vertical axes, ensuring that all position encodings are trained during the inference phase, thus improving resolution generalization. Additionally, we propose a random data augmentation technique to enhance the modeling of position order. To address the issue of image cropping caused by the augmentation, we introduce corresponding micro-conditioning to enable the model to perceive the specific cropping patterns. On the ImageNet dataset, our proposed RPE-2D achieves state-of-the-art resolution generalization performance, outperforming existing competitive methods when trained at a resolution of $256 \times 256$ and inferred at $384 \times 384$ and $512 \times 512$, as well as when scaling from $512 \times 512$ to $768 \times 768$ and $1024 \times 1024$. And it also exhibits outstanding capabilities in low-resolution image generation, multi-stage training acceleration and multi-resolution inheritance.