CodeNeRF: Disentangled Neural Radiance Fields for Object Categories
This work addresses the challenge of generalizing neural radiance fields to unseen objects in a category, which is useful for computer vision and graphics applications, though it builds incrementally on existing NeRF methods.
CodeNeRF tackles the problem of learning 3D neural representations for object categories from posed images, enabling novel view synthesis and editing of unseen objects from a single image, achieving on-par performance with methods requiring known camera pose on the SRN benchmark and bridging the sim-to-real gap.
CodeNeRF is an implicit 3D neural representation that learns the variation of object shapes and textures across a category and can be trained, from a set of posed images, to synthesize novel views of unseen objects. Unlike the original NeRF, which is scene specific, CodeNeRF learns to disentangle shape and texture by learning separate embeddings. At test time, given a single unposed image of an unseen object, CodeNeRF jointly estimates camera viewpoint, and shape and appearance codes via optimization. Unseen objects can be reconstructed from a single image, and then rendered from new viewpoints or their shape and texture edited by varying the latent codes. We conduct experiments on the SRN benchmark, which show that CodeNeRF generalises well to unseen objects and achieves on-par performance with methods that require known camera pose at test time. Our results on real-world images demonstrate that CodeNeRF can bridge the sim-to-real gap. Project page: \url{https://github.com/wayne1123/code-nerf}