FLAG: Flow-based 3D Avatar Generation from Sparse Observations
This addresses the challenge of creating faithful avatars for mixed reality collaboration, though it appears incremental as it builds on existing generative methods for human pose estimation.
The paper tackles the problem of generating realistic full-body 3D avatars from sparse head and hand pose observations, common in head-mounted devices, by developing a flow-based generative model that learns a conditional distribution of 3D human pose and provides uncertainty estimates for joints.
To represent people in mixed reality applications for collaboration and communication, we need to generate realistic and faithful avatar poses. However, the signal streams that can be applied for this task from head-mounted devices (HMDs) are typically limited to head pose and hand pose estimates. While these signals are valuable, they are an incomplete representation of the human body, making it challenging to generate a faithful full-body avatar. We address this challenge by developing a flow-based generative model of the 3D human body from sparse observations, wherein we learn not only a conditional distribution of 3D human pose, but also a probabilistic mapping from observations to the latent space from which we can generate a plausible pose along with uncertainty estimates for the joints. We show that our approach is not only a strong predictive model, but can also act as an efficient pose prior in different optimization settings where a good initial latent code plays a major role.