PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos
This addresses the issue of physically implausible motion estimates in computer vision for applications like action recognition, though it is incremental by building on existing kinematics methods.
The paper tackles the problem of physically unrealistic 3D human motion estimates from monocular videos by introducing PhysPT, which improves kinematics-based estimates and infers motion forces, leading to enhanced physical plausibility and improved accuracy in human action recognition.
While current methods have shown promising progress on estimating 3D human motion from monocular videos, their motion estimates are often physically unrealistic because they mainly consider kinematics. In this paper, we introduce Physics-aware Pretrained Transformer (PhysPT), which improves kinematics-based motion estimates and infers motion forces. PhysPT exploits a Transformer encoder-decoder backbone to effectively learn human dynamics in a self-supervised manner. Moreover, it incorporates physics principles governing human motion. Specifically, we build a physics-based body representation and contact force model. We leverage them to impose novel physics-inspired training losses (i.e., force loss, contact loss, and Euler-Lagrange loss), enabling PhysPT to capture physical properties of the human body and the forces it experiences. Experiments demonstrate that, once trained, PhysPT can be directly applied to kinematics-based estimates to significantly enhance their physical plausibility and generate favourable motion forces. Furthermore, we show that these physically meaningful quantities translate into improved accuracy of an important downstream task: human action recognition.