Controllable Shadow Generation with Single-Step Diffusion Models from Synthetic Data
This addresses the need for fast and controllable shadow generation in image compositing and visual effects, offering an incremental improvement over existing learning-based techniques.
The paper tackles the problem of realistic shadow generation for 2D object images by introducing a method that uses a diffusion model trained on synthetic data, achieving high-quality results with a single sampling step for real-time applications and demonstrating generalization to real-world images.
Realistic shadow generation is a critical component for high-quality image compositing and visual effects, yet existing methods suffer from certain limitations: Physics-based approaches require a 3D scene geometry, which is often unavailable, while learning-based techniques struggle with control and visual artifacts. We introduce a novel method for fast, controllable, and background-free shadow generation for 2D object images. We create a large synthetic dataset using a 3D rendering engine to train a diffusion model for controllable shadow generation, generating shadow maps for diverse light source parameters. Through extensive ablation studies, we find that rectified flow objective achieves high-quality results with just a single sampling step enabling real-time applications. Furthermore, our experiments demonstrate that the model generalizes well to real-world images. To facilitate further research in evaluating quality and controllability in shadow generation, we release a new public benchmark containing a diverse set of object images and shadow maps in various settings. The project page is available at https://gojasper.github.io/controllable-shadow-generation-project/