ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation
This work addresses the long-standing problem of 6-DoF pose tracking for challenging surfaces (textureless, transparent, reflective, deformable) without additional inputs, offering a new paradigm that leverages video pre-training.
ProxyPose introduces a video-to-video translation approach for 6-DoF pose tracking from monocular video, requiring only a single marked pixel in the first frame and no 3D models, depth, or masks. It achieves state-of-the-art accuracy on standard benchmarks after fine-tuning a video diffusion model on synthetic data, and extends to face tracking, camera pose estimation, and in-the-wild scenes.
Tracking the six-degree-of-freedom (6-DoF) pose of objects and surfaces from monocular video is a long-standing problem in computer vision. To tackle this problem, existing methods require inputs beyond the video itself-such as 3D models, depth maps, object masks, or task-specific learned features-and they struggle with textureless, transparent, reflective, or deformable surfaces. Here, we introduce ProxyPose, which recasts 6-DoF pose tracking as video-to-video translation. Given only a video and a single marked pixel in the first frame, a fine-tuned video diffusion model translates the input into a proxy video-a synthetic video depicting a colored polyhedron undergoing the same local rigid-body motion as the surface region at the marked pixel. Because the proxy's geometry and appearance are known by construction, recovering its full 6-DoF trajectory reduces to classical pose estimation with off-the-shelf solvers. This formulation leverages large-scale video pre-training to absorb the hardest aspects of pose tracking-handling challenging materials, occlusions, and deformations-into the translation step, while operating at the pixel level with no assumptions about object identity, boundaries, or global rigidity. ProxyPose achieves state-of-the-art 6-DoF pose tracking accuracy without the additional inputs required by competing methods and after fine-tuning the video model only on synthetic data. We further demonstrate that ProxyPose extends to face tracking, camera pose estimation, and challenging in-the-wild scenes that are beyond the reach of existing approaches. Project page: https://ruihangzhang97.github.io/proxypose/.