RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph
This addresses the data bottleneck in robotics for pose estimation, though it is incremental as it builds on existing methods by adding 3D priors.
The paper tackles the problem of estimating robot pose from monocular RGB images by proposing RoboTAG, which incorporates 3D priors and enables co-evolution of 2D and 3D representations to reduce reliance on labeled data, showing effectiveness across robot types.
Estimating robot pose from a monocular RGB image is a challenge in robotics and computer vision. Existing methods typically build networks on top of 2D visual backbones and depend heavily on labeled data for training, which is often scarce in real-world scenarios, causing a sim-to-real gap. Moreover, these approaches reduce the 3D-based problem to 2D domain, neglecting the 3D priors. To address these, we propose Robot Topological Alignment Graph (RoboTAG), which incorporates a 3D branch to inject 3D priors while enabling co-evolution of the 2D and 3D representations, alleviating the reliance on labels. Specifically, the RoboTAG consists of a 3D branch and a 2D branch, where nodes represent the states of the camera and robot system, and edges capture the dependencies between these variables or denote alignments between them. Closed loops are then defined in the graph, on which a consistency supervision across branches can be applied. This design allows us to utilize in-the-wild images as training data without annotations. Experimental results demonstrate that our method is effective across robot types, highlighting its potential to alleviate the data bottleneck in robotics.