InstructP2P: Learning to Edit 3D Point Clouds with Text Instructions
This work addresses the need for more intuitive and productive AI systems for 3D shape editing, though it is incremental as it builds on existing methods like Point-E and language models.
The paper tackles the problem of editing 3D point clouds using textual instructions by introducing InstructP2P, an end-to-end framework that enables color and geometry editing in a single forward pass, showing generalization to novel shape categories and instructions with limited training data.
Enhancing AI systems to perform tasks following human instructions can significantly boost productivity. In this paper, we present InstructP2P, an end-to-end framework for 3D shape editing on point clouds, guided by high-level textual instructions. InstructP2P extends the capabilities of existing methods by synergizing the strengths of a text-conditioned point cloud diffusion model, Point-E, and powerful language models, enabling color and geometry editing using language instructions. To train InstructP2P, we introduce a new shape editing dataset, constructed by integrating a shape segmentation dataset, off-the-shelf shape programs, and diverse edit instructions generated by a large language model, ChatGPT. Our proposed method allows for editing both color and geometry of specific regions in a single forward pass, while leaving other regions unaffected. In our experiments, InstructP2P shows generalization capabilities, adapting to novel shape categories and instructions, despite being trained on a limited amount of data.