Sketch Video Synthesis
This work addresses the problem of video-to-sketch conversion for applications in creative editing and animation, representing an incremental advancement in video processing techniques.
The paper tackles the challenge of generating temporally coherent sketches from videos by proposing an optimization-based framework using Bézier curves, achieving impressive visual abstraction and coherence as demonstrated in applications like sketch-based video editing.
Understanding semantic intricacies and high-level concepts is essential in image sketch generation, and this challenge becomes even more formidable when applied to the domain of videos. To address this, we propose a novel optimization-based framework for sketching videos represented by the frame-wise Bézier curve. In detail, we first propose a cross-frame stroke initialization approach to warm up the location and the width of each curve. Then, we optimize the locations of these curves by utilizing a semantic loss based on CLIP features and a newly designed consistency loss using the self-decomposed 2D atlas network. Built upon these design elements, the resulting sketch video showcases impressive visual abstraction and temporal coherence. Furthermore, by transforming a video into SVG lines through the sketching process, our method unlocks applications in sketch-based video editing and video doodling, enabled through video composition, as exemplified in the teaser.