JacobiNeRF: NeRF Shaping with Mutual Information Gradients
This method addresses the challenge of reducing annotation effort in 3D scene understanding by improving label propagation in NeRFs, though it is incremental as it builds on existing NeRF frameworks.
The paper tackles the problem of training neural radiance fields (NeRFs) to encode semantic correlations between scene entities by regularizing learning dynamics to align Jacobians, which maximizes mutual information under perturbations. The result is more efficient label propagation for semantic and instance segmentation, especially in sparse label regimes, reducing annotation burden.
We propose a method that trains a neural radiance field (NeRF) to encode not only the appearance of the scene but also semantic correlations between scene points, regions, or entities -- aiming to capture their mutual co-variation patterns. In contrast to the traditional first-order photometric reconstruction objective, our method explicitly regularizes the learning dynamics to align the Jacobians of highly-correlated entities, which proves to maximize the mutual information between them under random scene perturbations. By paying attention to this second-order information, we can shape a NeRF to express semantically meaningful synergies when the network weights are changed by a delta along the gradient of a single entity, region, or even a point. To demonstrate the merit of this mutual information modeling, we leverage the coordinated behavior of scene entities that emerges from our shaping to perform label propagation for semantic and instance segmentation. Our experiments show that a JacobiNeRF is more efficient in propagating annotations among 2D pixels and 3D points compared to NeRFs without mutual information shaping, especially in extremely sparse label regimes -- thus reducing annotation burden. The same machinery can further be used for entity selection or scene modifications.