Multi-view Image Diffusion via Coordinate Noise and Fourier Attention
This addresses the problem of multi-view consistency in text-to-image generation for applications like 3D scene synthesis, though it appears incremental as it builds on existing diffusion models.
The paper tackles the challenge of generating multi-view consistent images from text prompts using diffusion models, achieving state-of-the-art improvements on quantitative metrics with qualitatively better results.
Recently, text-to-image generation with diffusion models has made significant advancements in both higher fidelity and generalization capabilities compared to previous baselines. However, generating holistic multi-view consistent images from prompts still remains an important and challenging task. To address this challenge, we propose a diffusion process that attends to time-dependent spatial frequencies of features with a novel attention mechanism as well as novel noise initialization technique and cross-attention loss. This Fourier-based attention block focuses on features from non-overlapping regions of the generated scene in order to better align the global appearance. Our noise initialization technique incorporates shared noise and low spatial frequency information derived from pixel coordinates and depth maps to induce noise correlations across views. The cross-attention loss further aligns features sharing the same prompt across the scene. Our technique improves SOTA on several quantitative metrics with qualitatively better results when compared to other state-of-the-art approaches for multi-view consistency.