Text-Guided Scene Sketch-to-Photo Synthesis
This addresses the problem of generating realistic photos from sketches for artists or designers, but it is incremental as it builds on existing methods like Stable Diffusion.
The paper tackles scene-level sketch-to-photo synthesis by leveraging pre-trained generative models and self-supervised learning to translate hand-drawn sketches into photos without reference images, achieving compelling visual quality.
We propose a method for scene-level sketch-to-photo synthesis with text guidance. Although object-level sketch-to-photo synthesis has been widely studied, whole-scene synthesis is still challenging without reference photos that adequately reflect the target style. To this end, we leverage knowledge from recent large-scale pre-trained generative models, resulting in text-guided sketch-to-photo synthesis without the need for reference images. To train our model, we use self-supervised learning from a set of photographs. Specifically, we use a pre-trained edge detector that maps both color and sketch images into a standardized edge domain, which reduces the gap between photograph-based edge images (during training) and hand-drawn sketch images (during inference). We implement our method by fine-tuning a latent diffusion model (i.e., Stable Diffusion) with sketch and text conditions. Experiments show that the proposed method translates original sketch images that are not extracted from color images into photos with compelling visual quality.