InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation
This enables high-quality 3D human generation for applications like virtual reality and gaming, though it is an incremental improvement over existing methods.
The paper tackles the problem of generating photorealistic 3D human models with controllable inputs like text and poses, achieving state-of-the-art quality by addressing issues such as unnatural shading and limited views.
This paper presents InceptionHuman, a prompt-to-NeRF framework that allows easy control via a combination of prompts in different modalities (e.g., text, poses, edge, segmentation map, etc) as inputs to generate photorealistic 3D humans. While many works have focused on generating 3D human models, they suffer one or more of the following: lack of distinctive features, unnatural shading/shadows, unnatural poses/clothes, limited views, etc. InceptionHuman achieves consistent 3D human generation within a progressively refined NeRF space with two novel modules, Iterative Pose-Aware Refinement (IPAR) and Progressive-Augmented Reconstruction (PAR). IPAR iteratively refines the diffusion-generated images and synthesizes high-quality 3D-aware views considering the close-pose RGB values. PAR employs a pretrained diffusion prior to augment the generated synthetic views and adds regularization for view-independent appearance. Overall, the synthesis of photorealistic novel views empowers the resulting 3D human NeRF from 360-degree perspectives. Extensive qualitative and quantitative experimental comparison show that our InceptionHuman models achieve state-of-the-art application quality.