Canonical Fields: Self-Supervised Learning of Pose-Canonicalized Neural Fields
This addresses a problem in 3D computer vision for researchers and practitioners needing consistent object representations without manual alignment, though it is incremental as it builds on existing neural field techniques.
The paper tackles the challenge of building neural fields for object categories without consistently aligned datasets by introducing CaFi-Net, a self-supervised method that canonicalizes 3D pose from neural radiance fields, achieving performance that matches or exceeds 3D point cloud-based methods on a dataset of 1300 NeRF models across 13 categories.
Coordinate-based implicit neural networks, or neural fields, have emerged as useful representations of shape and appearance in 3D computer vision. Despite advances, however, it remains challenging to build neural fields for categories of objects without datasets like ShapeNet that provide "canonicalized" object instances that are consistently aligned for their 3D position and orientation (pose). We present Canonical Field Network (CaFi-Net), a self-supervised method to canonicalize the 3D pose of instances from an object category represented as neural fields, specifically neural radiance fields (NeRFs). CaFi-Net directly learns from continuous and noisy radiance fields using a Siamese network architecture that is designed to extract equivariant field features for category-level canonicalization. During inference, our method takes pre-trained neural radiance fields of novel object instances at arbitrary 3D pose and estimates a canonical field with consistent 3D pose across the entire category. Extensive experiments on a new dataset of 1300 NeRF models across 13 object categories show that our method matches or exceeds the performance of 3D point cloud-based methods.