Consistent Instance Field for Dynamic Scene Understanding
This work addresses the problem of consistent object representation in dynamic scenes for computer vision applications, offering a novel approach beyond incremental improvements.
The paper tackles dynamic scene understanding by introducing Consistent Instance Field, a continuous spatio-temporal representation that disentangles visibility from object identity, achieving significant improvements over state-of-the-art methods on novel-view panoptic segmentation and open-vocabulary 4D querying tasks.
We introduce Consistent Instance Field, a continuous and probabilistic spatio-temporal representation for dynamic scene understanding. Unlike prior methods that rely on discrete tracking or view-dependent features, our approach disentangles visibility from persistent object identity by modeling each space-time point with an occupancy probability and a conditional instance distribution. To realize this, we introduce a novel instance-embedded representation based on deformable 3D Gaussians, which jointly encode radiance and semantic information and are learned directly from input RGB images and instance masks through differentiable rasterization. Furthermore, we introduce new mechanisms to calibrate per-Gaussian identities and resample Gaussians toward semantically active regions, ensuring consistent instance representations across space and time. Experiments on HyperNeRF and Neu3D datasets demonstrate that our method significantly outperforms state-of-the-art methods on novel-view panoptic segmentation and open-vocabulary 4D querying tasks.