Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization
This addresses the need for more meaningful and controllable stylization in 3D graphics and vision applications, though it appears incremental as it builds on radiance fields and semantic techniques.
The paper tackles the problem of 3D scene stylization lacking semantic correspondences by introducing Ref-NPR, a method that uses a single stylized 2D view as a reference to achieve controllable stylization, outperforming existing methods in visual quality and semantic correspondence.
Current 3D scene stylization methods transfer textures and colors as styles using arbitrary style references, lacking meaningful semantic correspondences. We introduce Reference-Based Non-Photorealistic Radiance Fields (Ref-NPR) to address this limitation. This controllable method stylizes a 3D scene using radiance fields with a single stylized 2D view as a reference. We propose a ray registration process based on the stylized reference view to obtain pseudo-ray supervision in novel views. Then we exploit semantic correspondences in content images to fill occluded regions with perceptually similar styles, resulting in non-photorealistic and continuous novel view sequences. Our experimental results demonstrate that Ref-NPR outperforms existing scene and video stylization methods regarding visual quality and semantic correspondence. The code and data are publicly available on the project page at https://ref-npr.github.io.