Creative Robot Tool Use by Counterfactual Reasoning
This work addresses the problem of enabling robots to creatively use tools for tasks beyond their intended purpose, which is important for autonomous robots in unstructured environments.
The paper proposes a causal reasoning framework for creative robot tool use that identifies suitable tools beyond their primary objectives by discovering causal relationships between tool and task via simulated experiments in a dynamics model, achieving more reliable tool selection and stronger skill transfer compared to baselines.
We propose a causal reasoning framework for creative robot tool use where a suitable tool for a task is correctly identified for use beyond its primary objectives. The proposed framework first discovers the causal relationships between the tool and the task by conducting simulated experiments in a dynamics model. We decouple the causal discovery problem into two complementary components: VLM-based feature suggestion and counterfactual tool generation via targeted geometric and physical feature perturbations. Then, novel objects are classified based on identified causal features, and the tool use skill is transferred via keypoint matching conditioned on the identified causal features. By reconstructing the task in a dynamics model, our approach grounds tool use in the physics of the problem. We illustrate our approach in reaching a distant object with different sticks, scooping candies from a bowl using diverse items, and using different boxes or crates as stepping platforms to retrieve an object from a high shelf. Our baseline comparisons show that identifying causal features and grounding them in physical tool properties leads to more reliable tool selection and stronger skill keypoint transfer.