Beyond RGB: Scene-Property Synthesis with Neural Radiance Fields
This work addresses the problem of bridging generative and discriminative learning for 3D scene understanding, which could benefit applications like robot perception, though it appears incremental as it builds upon existing NeRF methods.
The paper tackles comprehensive 3D scene understanding by introducing SS-NeRF, a framework that extends Neural Radiance Fields to render not only RGB images but also accurate scene properties like geometry and semantics, enabling tasks such as semantic segmentation and surface normal estimation under a unified approach.
Comprehensive 3D scene understanding, both geometrically and semantically, is important for real-world applications such as robot perception. Most of the existing work has focused on developing data-driven discriminative models for scene understanding. This paper provides a new approach to scene understanding, from a synthesis model perspective, by leveraging the recent progress on implicit 3D representation and neural rendering. Building upon the great success of Neural Radiance Fields (NeRFs), we introduce Scene-Property Synthesis with NeRF (SS-NeRF) that is able to not only render photo-realistic RGB images from novel viewpoints, but also render various accurate scene properties (e.g., appearance, geometry, and semantics). By doing so, we facilitate addressing a variety of scene understanding tasks under a unified framework, including semantic segmentation, surface normal estimation, reshading, keypoint detection, and edge detection. Our SS-NeRF framework can be a powerful tool for bridging generative learning and discriminative learning, and thus be beneficial to the investigation of a wide range of interesting problems, such as studying task relationships within a synthesis paradigm, transferring knowledge to novel tasks, facilitating downstream discriminative tasks as ways of data augmentation, and serving as auto-labeller for data creation.