NPC: Neural Point Characters from Video
This addresses the need for more accessible and generalizable 3D character modeling from video, reducing reliance on costly capture systems, though it builds incrementally on prior neural representation techniques.
The paper tackles the problem of reconstructing animatable 3D characters from video without requiring expensive template surfaces, proposing a hybrid point-based representation that automatically extracts 3D points and learns articulated deformations, achieving performance matching methods using rigged surface templates on established benchmarks.
High-fidelity human 3D models can now be learned directly from videos, typically by combining a template-based surface model with neural representations. However, obtaining a template surface requires expensive multi-view capture systems, laser scans, or strictly controlled conditions. Previous methods avoid using a template but rely on a costly or ill-posed mapping from observation to canonical space. We propose a hybrid point-based representation for reconstructing animatable characters that does not require an explicit surface model, while being generalizable to novel poses. For a given video, our method automatically produces an explicit set of 3D points representing approximate canonical geometry, and learns an articulated deformation model that produces pose-dependent point transformations. The points serve both as a scaffold for high-frequency neural features and an anchor for efficiently mapping between observation and canonical space. We demonstrate on established benchmarks that our representation overcomes limitations of prior work operating in either canonical or in observation space. Moreover, our automatic point extraction approach enables learning models of human and animal characters alike, matching the performance of the methods using rigged surface templates despite being more general. Project website: https://lemonatsu.github.io/npc/