NF3DM: Combining Neural Fields and Deformation Models for 3D Non-Rigid Motion Reconstruction
This addresses the challenge of detailed and coherent 3D motion reconstruction for applications in animation or robotics, though it is incremental as it builds on existing neural field and deformation model techniques.
The paper tackles the problem of reconstructing temporally coherent 3D motion from unstructured and partial observations of non-rigidly deforming shapes, achieving high-fidelity results for near-isometric deformations like humans in loose clothing by combining neural fields and mesh-based deformation models, and it outperforms state-of-the-art approaches on human and animal motion sequences from monocular depth videos.
We introduce a novel, data-driven approach for reconstructing temporally coherent 3D motion from unstructured and potentially partial observations of non-rigidly deforming shapes. Our goal is to achieve high-fidelity motion reconstructions for shapes that undergo near-isometric deformations, such as humans wearing loose clothing. The key novelty of our work lies in its ability to combine implicit shape representations with explicit mesh-based deformation models, enabling detailed and temporally coherent motion reconstructions without relying on parametric shape models or decoupling shape and motion. Each frame is represented as a neural field decoded from a feature space where observations over time are fused, hence preserving geometric details present in the input data. Temporal coherence is enforced with a near-isometric deformation constraint between adjacent frames that applies to the underlying surface in the neural field. Our method outperforms state-of-the-art approaches, as demonstrated by its application to human and animal motion sequences reconstructed from monocular depth videos.