Instance Neural Radiance Field
This addresses the problem of 3D instance segmentation for scenes represented as NeRFs, enabling enhanced object segmentation and manipulation, though it is incremental as it builds on existing NeRF and segmentation techniques.
The paper tackles 3D instance segmentation in neural radiance fields (NeRF) by introducing Instance NeRF, which learns to segment instances from a pretrained NeRF model using 3D proposal-based mask prediction and 2D supervision from panoptic segmentation models, achieving superior segmentation performance on unseen views compared to previous NeRF and competitive 2D methods.
This paper presents one of the first learning-based NeRF 3D instance segmentation pipelines, dubbed as Instance Neural Radiance Field, or Instance NeRF. Taking a NeRF pretrained from multi-view RGB images as input, Instance NeRF can learn 3D instance segmentation of a given scene, represented as an instance field component of the NeRF model. To this end, we adopt a 3D proposal-based mask prediction network on the sampled volumetric features from NeRF, which generates discrete 3D instance masks. The coarse 3D mask prediction is then projected to image space to match 2D segmentation masks from different views generated by existing panoptic segmentation models, which are used to supervise the training of the instance field. Notably, beyond generating consistent 2D segmentation maps from novel views, Instance NeRF can query instance information at any 3D point, which greatly enhances NeRF object segmentation and manipulation. Our method is also one of the first to achieve such results in pure inference. Experimented on synthetic and real-world NeRF datasets with complex indoor scenes, Instance NeRF surpasses previous NeRF segmentation works and competitive 2D segmentation methods in segmentation performance on unseen views. Watch the demo video at https://youtu.be/wW9Bme73coI. Code and data are available at https://github.com/lyclyc52/Instance_NeRF.