Correspondence Distillation from NeRF-based GAN
This work addresses a key limitation in NeRF representations for downstream tasks like texture transfer, though it is incremental as it builds on existing NeRF-based GANs.
The paper tackles the problem of building dense correspondences across different NeRF models of the same category, which is challenging due to NeRF's implicit nature and lack of ground-truth annotations, and achieves accurate, smooth, and robust 3D dense correspondence by leveraging priors from a pre-trained NeRF-based GAN.
The neural radiance field (NeRF) has shown promising results in preserving the fine details of objects and scenes. However, unlike mesh-based representations, it remains an open problem to build dense correspondences across different NeRFs of the same category, which is essential in many downstream tasks. The main difficulties of this problem lie in the implicit nature of NeRF and the lack of ground-truth correspondence annotations. In this paper, we show it is possible to bypass these challenges by leveraging the rich semantics and structural priors encapsulated in a pre-trained NeRF-based GAN. Specifically, we exploit such priors from three aspects, namely 1) a dual deformation field that takes latent codes as global structural indicators, 2) a learning objective that regards generator features as geometric-aware local descriptors, and 3) a source of infinite object-specific NeRF samples. Our experiments demonstrate that such priors lead to 3D dense correspondence that is accurate, smooth, and robust. We also show that established dense correspondence across NeRFs can effectively enable many NeRF-based downstream applications such as texture transfer.