From Grasps to Dexterity: Large-Scale Grasp Pretraining for Dexterous Manipulation
This work addresses the problem of leveraging grasp data for contact-rich dexterous manipulation, showing that grasp datasets can serve as scalable pretraining data beyond grasp synthesis.
The paper investigates whether large-scale dexterous grasp datasets can be used for pretraining to enable functional dexterity in articulated tool use. The proposed hierarchical imitation learning approach, pretrained on a 355k-trajectory grasp dataset and fine-tuned on task demonstrations, achieves a 33.3 percentage point improvement in full-task success over DP3 in real-world experiments.
Large-scale dexterous grasp datasets encode rich priors over hand-object interaction, but their use has largely been confined to grasp generation and pick-and-place manipulation. We study whether such data can instead support functional dexterity in articulated tool use, where a robot must acquire a tool, maintain contact, and operate its functional moving parts. We adapt a hierarchical imitation learning framework that combines high-level hand sub-goal prediction with a low-level goal-conditioned controller. We construct a 355k-trajectory grasp-pretraining dataset from large-scale dexterous grasp annotations and use it to pretrain the low-level controller. The controller is then fine-tuned on downstream task demonstrations. To evaluate this setting, we introduce DexCraft, a simulation benchmark with six articulated tool-use tasks requiring coordinated finger motion. Across simulation and real-world experiments, our approach outperforms end-to-end diffusion policy baselines and hierarchical policies trained from scratch. In the real world, it improves full-task success by 33.3 percentage points over DP3. These results show that grasp datasets can serve not only as resources for grasp synthesis, but also as scalable pretraining data for contact-rich dexterous manipulation. Videos are shown on https://yingyuan0414.github.io/grasp2dexterity/ .