Keypose Exploration: Efficient Automatic Trajectory Labelling and Cross-Embodiment Policy Transfer
This work addresses the need for automatic keypose extraction in manipulation tasks, reducing manual annotation effort, but results are preliminary and limited to two simulated tasks.
The paper introduces an automatic trajectory labelling pipeline for grasp-related tasks that combines vision-language models with classical trajectory analysis, requiring VLM inference on only one demonstration per task. Using the labelled data, a keypose-guided Diffusion Policy is trained, and a reachability map is used to filter candidate keyposes for zero-shot cross-embodiment transfer, showing potential benefits on a multimodal insertion task.
Keypose-based manipulation decomposes tasks into critical waypoints to simplify policy learning for long-horizon tasks, but existing approaches rely on task-specific heuristics or manual annotation to extract keyposes from demonstrations. We present an automatic trajectory labelling pipeline for grasp-related tasks. This pipeline combines vision-language models (VLMs) for semantic event detection with classical trajectory analysis for precise temporal alignment, requiring VLM inference only on one single demo among repeating ones per task. Using the labelled data, we train a keypose-guided Diffusion Policy (DP) that exploits keypose conditioning to intervene demonstration distributions. We explore the possibility to apply this property for cross-embodiment transfer: candidate keyposes are sampled and filtered via a reachability map, steering the policy toward kinematically feasible keyposes for the target robot. As a preliminary feasibility study, experiments on two robomimic tasks show that the labelled data produces policies matching a standard DP baseline, and that reachability-filtered keypose conditioning may benefit zero-shot transfer on the multimodal insertion task when feasible candidates are available.