RoboHitch: Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying
For robotic manipulation of deformable linear objects, this work addresses the bottleneck of tracking drift and topology mismatch in knot tying by removing the need for ordered keypoints.
RoboHitch learns to tie hitch knots from human demonstrations using only disordered 3D keypoints and RGB images, eliminating the need for explicit topological tracking. Real-world experiments show successful knot tying under self-occlusions, with no specific performance numbers reported.
Robotic manipulation of deformable linear objects (DLOs) presents significant challenges due to complex dynamics and frequent self-occlusions. Existing robotic knot tying methods typically rely on precise topological state tracking with ordered keypoints and explicit edge connectivity. This reliance makes them prone to failures due to tracking drift and topology mismatch caused by repeated bending and crossings during knot formation.To address these limitations, we introduce RoboHitch, a novel framework that learns to perform hitch knot tying from human demonstrations using only disordered 3D keypoints and RGB images. This eliminates the need for explicit topological order, allowing for more flexible manipulation. Our method employs a dynamic Graph Autoencoder to extract geometric features from untracked keypoints, complemented by a Convolutional Autoencoder that captures essential visual context. A bidirectional cross-attention mechanism then fuses these modalities to jointly predict pick and place affordances, facilitating implicit reasoning about the rope's state and enabling knot tying under occlusion.Real-world experiments demonstrate the effectiveness and generalizability of our approach, successfully completing hitch knots in scenarios with self-occlusions.