DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models
This addresses the challenge of creating abstract yet recognizable vector sketches for artists and designers, though it is incremental as it builds on existing diffusion models.
The paper tackles the problem of generating vectorized free-hand sketches from natural language input by using a pre-trained text-to-image diffusion model to optimize Bézier curves with an extended score distillation sampling loss, achieving greater quality than prior work.
Even though trained mainly on images, we discover that pretrained diffusion models show impressive power in guiding sketch synthesis. In this paper, we present DiffSketcher, an innovative algorithm that creates \textit{vectorized} free-hand sketches using natural language input. DiffSketcher is developed based on a pre-trained text-to-image diffusion model. It performs the task by directly optimizing a set of Bézier curves with an extended version of the score distillation sampling (SDS) loss, which allows us to use a raster-level diffusion model as a prior for optimizing a parametric vectorized sketch generator. Furthermore, we explore attention maps embedded in the diffusion model for effective stroke initialization to speed up the generation process. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual details of the subject drawn. Our experiments show that DiffSketcher achieves greater quality than prior work. The code and demo of DiffSketcher can be found at https://ximinng.github.io/DiffSketcher-project/.