LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation
This addresses the challenge of inefficient and computationally heavy lighting control in image generation for users of diffusion models, though it is incremental as it builds on existing noise manipulation techniques.
The paper tackles the problem of controlling lighting conditions in text-to-image diffusion models, proposing a training-free method that manipulates initial latent noise to achieve fine-grained lighting control, surpassing prompt-based baselines in lighting consistency while preserving image quality and text alignment.
Diffusion models have demonstrated high-quality performance in conditional text-to-image generation, particularly with structural cues such as edges, layouts, and depth. However, lighting conditions have received limited attention and remain difficult to control within the generative process. Existing methods handle lighting through a two-stage pipeline that relights images after generation, which is inefficient. Moreover, they rely on fine-tuning with large datasets and heavy computation, limiting their adaptability to new models and tasks. To address this, we propose a novel Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation (LGTM), which manipulates the initial latent noise of the diffusion process to guide image generation with text prompts and user-specified light directions. Through a channel-wise analysis of the latent space, we find that selectively manipulating latent channels enables fine-grained lighting control without fine-tuning or modifying the pre-trained model. Extensive experiments show that our method surpasses prompt-based baselines in lighting consistency, while preserving image quality and text alignment. This approach introduces new possibilities for dynamic, user-guided light control. Furthermore, it integrates seamlessly with models like ControlNet, demonstrating adaptability across diverse scenarios.