PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields
This addresses the challenge of appearance editing in 3D scene representations for applications in computer graphics and vision, though it is incremental as it builds on existing NeRF methods.
The paper tackles the problem of efficiently editing the appearance of neural radiance fields while maintaining photorealism, and presents PaletteNeRF, which enables photorealistic appearance editing through 3D color decomposition and palette-based modifications, showing superiority over baseline methods quantitatively and qualitatively.
Recent advances in neural radiance fields have enabled the high-fidelity 3D reconstruction of complex scenes for novel view synthesis. However, it remains underexplored how the appearance of such representations can be efficiently edited while maintaining photorealism. In this work, we present PaletteNeRF, a novel method for photorealistic appearance editing of neural radiance fields (NeRF) based on 3D color decomposition. Our method decomposes the appearance of each 3D point into a linear combination of palette-based bases (i.e., 3D segmentations defined by a group of NeRF-type functions) that are shared across the scene. While our palette-based bases are view-independent, we also predict a view-dependent function to capture the color residual (e.g., specular shading). During training, we jointly optimize the basis functions and the color palettes, and we also introduce novel regularizers to encourage the spatial coherence of the decomposition. Our method allows users to efficiently edit the appearance of the 3D scene by modifying the color palettes. We also extend our framework with compressed semantic features for semantic-aware appearance editing. We demonstrate that our technique is superior to baseline methods both quantitatively and qualitatively for appearance editing of complex real-world scenes.