Order Matters: 3D Shape Generation from Sequential VR Sketches
This work solves the problem of more intuitive and accurate 3D shape generation from VR sketches for designers and creators, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackles the problem of generating 3D shapes from VR sketches by addressing the limitation of existing models that ignore stroke order, introducing VRSketch2Shape with a dataset of over 20k synthetic and 900 hand-drawn sketch-shape pairs, resulting in higher geometric fidelity and effective generalization to real sketches with minimal supervision.
VR sketching lets users explore and iterate on ideas directly in 3D, offering a faster and more intuitive alternative to conventional CAD tools. However, existing sketch-to-shape models ignore the temporal ordering of strokes, discarding crucial cues about structure and design intent. We introduce VRSketch2Shape, the first framework and multi-category dataset for generating 3D shapes from sequential VR sketches. Our contributions are threefold: (i) an automated pipeline that generates sequential VR sketches from arbitrary shapes, (ii) a dataset of over 20k synthetic and 900 hand-drawn sketch-shape pairs across four categories, and (iii) an order-aware sketch encoder coupled with a diffusion-based 3D generator. Our approach yields higher geometric fidelity than prior work, generalizes effectively from synthetic to real sketches with minimal supervision, and performs well even on partial sketches. All data and models will be released open-source at https://chenyizi086.github.io/VRSketch2Shape_website.