Canonical Factors for Hybrid Neural Fields
This addresses efficiency and bias issues in neural fields for 3D reconstruction tasks, offering an incremental improvement over existing factored methods.
The paper tackles biases in factored feature volumes for neural fields, showing they cause up to 2 PSNR differences in radiance field reconstruction, and introduces TILTED with canonicalizing transformations to improve quality, robustness, compactness, and runtime, enabling comparable performance to baselines twice as large.
Factored feature volumes offer a simple way to build more compact, efficient, and intepretable neural fields, but also introduce biases that are not necessarily beneficial for real-world data. In this work, we (1) characterize the undesirable biases that these architectures have for axis-aligned signals -- they can lead to radiance field reconstruction differences of as high as 2 PSNR -- and (2) explore how learning a set of canonicalizing transformations can improve representations by removing these biases. We prove in a two-dimensional model problem that simultaneously learning these transformations together with scene appearance succeeds with drastically improved efficiency. We validate the resulting architectures, which we call TILTED, using image, signed distance, and radiance field reconstruction tasks, where we observe improvements across quality, robustness, compactness, and runtime. Results demonstrate that TILTED can enable capabilities comparable to baselines that are 2x larger, while highlighting weaknesses of neural field evaluation procedures.