ProcessPainter: Learn Painting Process from Sequence Data
This work addresses the need for authentic painting process generation in art education and image generation, though it appears incremental as it builds on existing text-to-video and fine-tuning techniques.
The paper tackles the problem of generating step-by-step painting processes from text prompts, which is underexplored for art education and research, and introduces ProcessPainter, a text-to-video model that successfully achieves this for the first time by pre-training on synthetic data and fine-tuning with artists' sequences using LoRA.
The painting process of artists is inherently stepwise and varies significantly among different painters and styles. Generating detailed, step-by-step painting processes is essential for art education and research, yet remains largely underexplored. Traditional stroke-based rendering methods break down images into sequences of brushstrokes, yet they fall short of replicating the authentic processes of artists, with limitations confined to basic brushstroke modifications. Text-to-image models utilizing diffusion processes generate images through iterative denoising, also diverge substantially from artists' painting process. To address these challenges, we introduce ProcessPainter, a text-to-video model that is initially pre-trained on synthetic data and subsequently fine-tuned with a select set of artists' painting sequences using the LoRA model. This approach successfully generates painting processes from text prompts for the first time. Furthermore, we introduce an Artwork Replication Network capable of accepting arbitrary-frame input, which facilitates the controlled generation of painting processes, decomposing images into painting sequences, and completing semi-finished artworks. This paper offers new perspectives and tools for advancing art education and image generation technology.