Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision
This addresses the problem of accurate body pose and shape estimation from wearable cameras for applications like VR/AR, though it is incremental as it builds on existing HMR techniques with a novel adaptation for fisheye distortions.
The paper tackles 3D human mesh recovery from egocentric fisheye images by introducing Fish2Mesh, a transformer-based model with egocentric position embeddings, which outperforms previous state-of-the-art models in experiments.
Egocentric human body estimation allows for the inference of user body pose and shape from a wearable camera's first-person perspective. Although research has used pose estimation techniques to overcome self-occlusions and image distortions caused by head-mounted fisheye images, similar advances in 3D human mesh recovery (HMR) techniques have been limited. We introduce Fish2Mesh, a fisheye-aware transformer-based model designed for 3D egocentric human mesh recovery. We propose an egocentric position embedding block to generate an ego-specific position table for the Swin Transformer to reduce fisheye image distortion. Our model utilizes multi-task heads for SMPL parametric regression and camera translations, estimating 3D and 2D joints as auxiliary loss to support model training. To address the scarcity of egocentric camera data, we create a training dataset by employing the pre-trained 4D-Human model and third-person cameras for weak supervision. Our experiments demonstrate that Fish2Mesh outperforms previous state-of-the-art 3D HMR models.