ScanEdit: Hierarchically-Guided Functional 3D Scan Editing
This work addresses the need for effective 3D scene editing in graphics applications, offering a novel approach for handling real-world scans, though it is incremental in combining existing techniques like LLMs with hierarchical structures.
The authors tackled the problem of editing complex 3D scans by introducing ScanEdit, an instruction-driven method that uses hierarchical scene graphs and LLMs to translate language instructions into actionable commands, outperforming state-of-the-art methods in experimental evaluations.
With the fast pace of 3D capture technology and resulting abundance of 3D data, effective 3D scene editing becomes essential for a variety of graphics applications. In this work we present ScanEdit, an instruction-driven method for functional editing of complex, real-world 3D scans. To model large and interdependent sets of ob- jectswe propose a hierarchically-guided approach. Given a 3D scan decomposed into its object instances, we first construct a hierarchical scene graph representation to enable effective, tractable editing. We then leverage reason- ing capabilities of Large Language Models (LLMs) and translate high-level language instructions into actionable commands applied hierarchically to the scene graph. Fi- nally, ScanEdit integrates LLM-based guidance with ex- plicit physical constraints and generates realistic scenes where object arrangements obey both physics and common sense. In our extensive experimental evaluation ScanEdit outperforms state of the art and demonstrates excellent re- sults for a variety of real-world scenes and input instruc- tions.