Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing
This work addresses the need for graphics applications to manipulate object appearance and physics, offering a novel integration of language-driven semantics with 3D scene synthesis.
The paper tackles the problem of synthesizing and editing 3D scenes with both appearance and physical properties by introducing Feature Splatting, which unifies physics-based dynamics with semantics from vision-language models, enabling semi-automatic scene decomposition and material assignment via text queries.
Scene representations using 3D Gaussian primitives have produced excellent results in modeling the appearance of static and dynamic 3D scenes. Many graphics applications, however, demand the ability to manipulate both the appearance and the physical properties of objects. We introduce Feature Splatting, an approach that unifies physics-based dynamic scene synthesis with rich semantics from vision language foundation models that are grounded by natural language. Our first contribution is a way to distill high-quality, object-centric vision-language features into 3D Gaussians, that enables semi-automatic scene decomposition using text queries. Our second contribution is a way to synthesize physics-based dynamics from an otherwise static scene using a particle-based simulator, in which material properties are assigned automatically via text queries. We ablate key techniques used in this pipeline, to illustrate the challenge and opportunities in using feature-carrying 3D Gaussians as a unified format for appearance, geometry, material properties and semantics grounded on natural language. Project website: https://feature-splatting.github.io/