Video Inference for Human Mesh Recovery with Vision Transformer
This addresses the challenge of ambiguity in HMR for computer vision applications, but it is incremental as it combines existing approaches without a major breakthrough.
The paper tackles the problem of Human Mesh Recovery (HMR) from images by proposing HMR-ViT, a method that integrates both temporal and kinematic information using a Vision Transformer, achieving competitive performance on datasets like 3DPW and Human3.6M.
Human Mesh Recovery (HMR) from an image is a challenging problem because of the inherent ambiguity of the task. Existing HMR methods utilized either temporal information or kinematic relationships to achieve higher accuracy, but there is no method using both. Hence, we propose "Video Inference for Human Mesh Recovery with Vision Transformer (HMR-ViT)" that can take into account both temporal and kinematic information. In HMR-ViT, a Temporal-kinematic Feature Image is constructed using feature vectors obtained from video frames by an image encoder. When generating the feature image, we use a Channel Rearranging Matrix (CRM) so that similar kinematic features could be located spatially close together. The feature image is then further encoded using Vision Transformer, and the SMPL pose and shape parameters are finally inferred using a regression network. Extensive evaluation on the 3DPW and Human3.6M datasets indicates that our method achieves a competitive performance in HMR.