MS2Mesh-XR: Multi-modal Sketch-to-Mesh Generation in XR Environments
This work addresses the challenge of immersive 3D content creation for XR users, offering a practical tool for creative production, though it appears incremental by integrating existing AI components.
The paper tackles the problem of creating realistic 3D objects in extended reality (XR) environments by introducing MS2Mesh-XR, a multi-modal sketch-to-mesh generation pipeline that uses hand-drawn sketches and voice inputs to generate high-quality 3D meshes in less than 20 seconds.
We present MS2Mesh-XR, a novel multi-modal sketch-to-mesh generation pipeline that enables users to create realistic 3D objects in extended reality (XR) environments using hand-drawn sketches assisted by voice inputs. In specific, users can intuitively sketch objects using natural hand movements in mid-air within a virtual environment. By integrating voice inputs, we devise ControlNet to infer realistic images based on the drawn sketches and interpreted text prompts. Users can then review and select their preferred image, which is subsequently reconstructed into a detailed 3D mesh using the Convolutional Reconstruction Model. In particular, our proposed pipeline can generate a high-quality 3D mesh in less than 20 seconds, allowing for immersive visualization and manipulation in run-time XR scenes. We demonstrate the practicability of our pipeline through two use cases in XR settings. By leveraging natural user inputs and cutting-edge generative AI capabilities, our approach can significantly facilitate XR-based creative production and enhance user experiences. Our code and demo will be available at: https://yueqiu0911.github.io/MS2Mesh-XR/