VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Control
This enables ordinary users to create video animations from sketches, bridging a gap with professional artists, but it is incremental as it builds on existing sketch-to-image and video generation techniques.
The paper tackles the problem of generating video animations from hand-drawn sketches, which previous methods limited to static images, and achieves high-quality results with improved spatiotemporal consistency.
With the advancement of generative artificial intelligence, previous studies have achieved the task of generating aesthetic images from hand-drawn sketches, fulfilling the public's needs for drawing. However, these methods are limited to static images and lack the ability to control video animation generation using hand-drawn sketches. To address this gap, we propose VidSketch, the first method capable of generating high-quality video animations directly from any number of hand-drawn sketches and simple text prompts, bridging the divide between ordinary users and professional artists. Specifically, our method introduces a Level-Based Sketch Control Strategy to automatically adjust the guidance strength of sketches during the generation process, accommodating users with varying drawing skills. Furthermore, a TempSpatial Attention mechanism is designed to enhance the spatiotemporal consistency of generated video animations, significantly improving the coherence across frames. You can find more detailed cases on our official website.