MvDrag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Priors
This work addresses the problem of creative 3D content editing for users in graphics and AI, offering a more flexible solution than existing methods, though it appears incremental as it builds on prior 2D and 3D techniques.
The paper tackles the challenge of extending drag-based editing from 2D to 3D by introducing MVDrag3D, a framework that uses multi-view generation and reconstruction priors to enable flexible 3D editing with support for topology changes and new textures across diverse object categories, achieving precise and versatile editing effects.
Drag-based editing has become popular in 2D content creation, driven by the capabilities of image generative models. However, extending this technique to 3D remains a challenge. Existing 3D drag-based editing methods, whether employing explicit spatial transformations or relying on implicit latent optimization within limited-capacity 3D generative models, fall short in handling significant topology changes or generating new textures across diverse object categories. To overcome these limitations, we introduce MVDrag3D, a novel framework for more flexible and creative drag-based 3D editing that leverages multi-view generation and reconstruction priors. At the core of our approach is the usage of a multi-view diffusion model as a strong generative prior to perform consistent drag editing over multiple rendered views, which is followed by a reconstruction model that reconstructs 3D Gaussians of the edited object. While the initial 3D Gaussians may suffer from misalignment between different views, we address this via view-specific deformation networks that adjust the position of Gaussians to be well aligned. In addition, we propose a multi-view score function that distills generative priors from multiple views to further enhance the view consistency and visual quality. Extensive experiments demonstrate that MVDrag3D provides a precise, generative, and flexible solution for 3D drag-based editing, supporting more versatile editing effects across various object categories and 3D representations.