Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views
This addresses the problem of 3D human mesh recovery in AR/VR applications for social interaction scenarios, representing a novel method for a known bottleneck.
The paper tackles the challenge of estimating 3D human pose and shape from egocentric views, where severe body truncation causes pose ambiguities, by proposing a scene-conditioned diffusion method that generates plausible human-scene interactions with physics-based guidance. It achieves superior accuracy for visible joints and diversity for invisible body parts, with code made publicly available.
Automatic perception of human behaviors during social interactions is crucial for AR/VR applications, and an essential component is estimation of plausible 3D human pose and shape of our social partners from the egocentric view. One of the biggest challenges of this task is severe body truncation due to close social distances in egocentric scenarios, which brings large pose ambiguities for unseen body parts. To tackle this challenge, we propose a novel scene-conditioned diffusion method to model the body pose distribution. Conditioned on the 3D scene geometry, the diffusion model generates bodies in plausible human-scene interactions, with the sampling guided by a physics-based collision score to further resolve human-scene inter-penetrations. The classifier-free training enables flexible sampling with different conditions and enhanced diversity. A visibility-aware graph convolution model guided by per-joint visibility serves as the diffusion denoiser to incorporate inter-joint dependencies and per-body-part control. Extensive evaluations show that our method generates bodies in plausible interactions with 3D scenes, achieving both superior accuracy for visible joints and diversity for invisible body parts. The code is available at https://sanweiliti.github.io/egohmr/egohmr.html.