ASSIST: Interactive Scene Nodes for Scalable and Realistic Indoor Simulation
This addresses the need for user-friendly, scalable simulation tools in fields like robotics and virtual reality, though it appears incremental as it builds on existing neural radiance field methods.
The paper tackles the problem of scalable and realistic indoor simulation by introducing ASSIST, an object-wise neural radiance field that enables interactive editing and compositional rendering, resulting in the generation of color images, depth images, and panoptic segmentation masks in a 3D consistent manner.
We present ASSIST, an object-wise neural radiance field as a panoptic representation for compositional and realistic simulation. Central to our approach is a novel scene node data structure that stores the information of each object in a unified fashion, allowing online interaction in both intra- and cross-scene settings. By incorporating a differentiable neural network along with the associated bounding box and semantic features, the proposed structure guarantees user-friendly interaction on independent objects to scale up novel view simulation. Objects in the scene can be queried, added, duplicated, deleted, transformed, or swapped simply through mouse/keyboard controls or language instructions. Experiments demonstrate the efficacy of the proposed method, where scaled realistic simulation can be achieved through interactive editing and compositional rendering, with color images, depth images, and panoptic segmentation masks generated in a 3D consistent manner.