Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions
This addresses the challenge of reducing development time for deployable tactile skills in robotics, though it is incremental by applying LLMs to a new domain with tactile sensing.
The paper tackles the problem of automating reward design for dexterous manipulation using LLMs, specifically for multi-axis in-hand object rotation with tactile sensing, and shows that Text2Touch outperforms a human-engineered baseline with superior rotation speed and stability using simpler reward functions.
Large language models (LLMs) are beginning to automate reward design for dexterous manipulation. However, no prior work has considered tactile sensing, which is known to be critical for human-like dexterity. We present Text2Touch, bringing LLM-crafted rewards to the challenging task of multi-axis in-hand object rotation with real-world vision based tactile sensing in palm-up and palm-down configurations. Our prompt engineering strategy scales to over 70 environment variables, and sim-to-real distillation enables successful policy transfer to a tactile-enabled fully actuated four-fingered dexterous robot hand. Text2Touch significantly outperforms a carefully tuned human-engineered baseline, demonstrating superior rotation speed and stability while relying on reward functions that are an order of magnitude shorter and simpler. These results illustrate how LLM-designed rewards can significantly reduce the time from concept to deployable dexterous tactile skills, supporting more rapid and scalable multimodal robot learning. Project website: https://hpfield.github.io/text2touch-website