FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning
For robotic manipulation, this work addresses the critical bottleneck of generalizing tool use to novel objects, enabling robots to repurpose everyday objects for tasks like hammering.
The paper tackles the problem of functional generalization in robotic tool use, where robots fail to transfer learned tool functions to novel tools. The proposed FORGE method, which decouples functional reasoning (keypoint trajectory prediction) from action execution, achieves over 2X improvement in average success rate on unseen tools compared to state-of-the-art methods.
While humans readily repurpose a book, a stone, or a shoe to drive a nail, robots trained on specific tools fail to transfer the same function to novel ones -- a gap we formalize as functional generalization. Such tools share a common functional intent that is visually recognizable, yet this perceptual similarity does not carry over to action space, where each tool demands an entirely different motor pattern. To bridge this gap, we explore intermediate representations including affordance images, human video prompts, and 2D keypoint trajectories, finding that keypoint trajectories best balance functional expressiveness and action groundability. Building on this, we propose FunctiOnal Reasoning and Grounded Execution (FORGE), a two-stage policy that decouples functional reasoning from action execution: predicting generalizable keypoint trajectories from action-free data, then grounding them into robot actions with limited demonstrations. On a seven-tool hitting-function benchmark, FORGE consistently outperforms state-of-the-art methods on unseen tools in both simulation and the real world, achieving over 2X improvement in average success rate.