${M^2D}$NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields
This work addresses the challenge of scene decomposition in NeRF for applications like 3D editing, but it is incremental as it builds on existing multi-modal and NeRF techniques.
The paper tackles the problem of semantic encoding in Neural Radiance Fields (NeRF) for 3D scene decomposition by proposing M^2DNeRF, a model that integrates multi-modal features from visual and language models into 3D semantic volumes, enabling text-based and visual patch-based edits, and it shows superior performance in experiments on real-world scenes compared to prior methods.
Neural fields (NeRF) have emerged as a promising approach for representing continuous 3D scenes. Nevertheless, the lack of semantic encoding in NeRFs poses a significant challenge for scene decomposition. To address this challenge, we present a single model, Multi-Modal Decomposition NeRF (${M^2D}$NeRF), that is capable of both text-based and visual patch-based edits. Specifically, we use multi-modal feature distillation to integrate teacher features from pretrained visual and language models into 3D semantic feature volumes, thereby facilitating consistent 3D editing. To enforce consistency between the visual and language features in our 3D feature volumes, we introduce a multi-modal similarity constraint. We also introduce a patch-based joint contrastive loss that helps to encourage object-regions to coalesce in the 3D feature space, resulting in more precise boundaries. Experiments on various real-world scenes show superior performance in 3D scene decomposition tasks compared to prior NeRF-based methods.