Multi-HMR 2: Multi-Person Camera-Centric Human Detection, Mesh Recovery and Tracking
This work addresses the need for robust metric 3D localization and tracking in multi-person human mesh recovery, which is critical for real-world applications like human-robot interaction.
Multi-HMR 2 introduces a DETR-based framework for multi-person camera-centric human mesh recovery that predicts a scene-consistent camera and uses SAM2 memory features for tracking, achieving state-of-the-art pelvis-centered performance while significantly improving detection accuracy and metric 3D localization.
Most advances in human mesh recovery (HMR) have focused on pelvis-centered recovery, overlooking metric 3D localization and detection accuracy in the camera coordinate system - two key factors for real-world applications such as human-robot interaction and social scene understanding. Current evaluation protocols often ignore these aspects, emphasizing per-person, root-centered recovery rather than camera-space perception. As a result, existing approaches rely on fixed camera assumptions or handcrafted post-processing, limiting their robustness and practical deployment. We introduce Multi-HMR 2, a simple yet robust DETR-based framework for Multi-person Camera-centric Human detection, mesh Recovery, and tracking. Multi-HMR 2 predicts a scene-consistent camera together with human meshes, enabling metric 3D localization without ground-truth intrinsics. Moreover, by distilling image-based memory features from SAM2, Multi-HMR 2 extends to tracking, achieving consistent identity association without video supervision. Despite its conceptual simplicity - no handcrafted components, no video input, and no ground-truth cameras - Multi-HMR 2 achieves state-of-the-art pelvis-centered performance while substantially improving detection accuracy and metric 3D localization.