DFU: scale-robust diffusion model for zero-shot super-resolution image generation
This addresses a scalability limitation in diffusion models for image generation, enabling coherent high-resolution outputs without resolution-specific training, though it appears incremental in combining existing techniques.
The paper tackles the problem of diffusion models' inability to generalize to different resolutions without training data at those resolutions, and presents DFU, a novel architecture that achieves zero-shot super-resolution image generation with a FID of 11.3 at 1.66 times the maximum training resolution on FFHQ.
Diffusion generative models have achieved remarkable success in generating images with a fixed resolution. However, existing models have limited ability to generalize to different resolutions when training data at those resolutions are not available. Leveraging techniques from operator learning, we present a novel deep-learning architecture, Dual-FNO UNet (DFU), which approximates the score operator by combining both spatial and spectral information at multiple resolutions. Comparisons of DFU to baselines demonstrate its scalability: 1) simultaneously training on multiple resolutions improves FID over training at any single fixed resolution; 2) DFU generalizes beyond its training resolutions, allowing for coherent, high-fidelity generation at higher-resolutions with the same model, i.e. zero-shot super-resolution image-generation; 3) we propose a fine-tuning strategy to further enhance the zero-shot super-resolution image-generation capability of our model, leading to a FID of 11.3 at 1.66 times the maximum training resolution on FFHQ, which no other method can come close to achieving.