Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation with Large Language Models
This work addresses the problem of reward function design for robotics researchers and practitioners, offering an incremental improvement by automating a known bottleneck in reinforcement learning.
The paper tackles the challenge of designing effective reward functions for reinforcement learning in real-world robotic manipulation by proposing ARCHIE, an unsupervised pipeline that uses GPT-4 to generate reward functions from natural language task descriptions, resulting in a fully automated procedure validated through simulated experiments and real robot demonstrations.
Recent advancements in Large Language Models (LLMs) and Visual Language Models (VLMs) have significantly impacted robotics, enabling high-level semantic motion planning applications. Reinforcement Learning (RL), a complementary paradigm, enables agents to autonomously optimize complex behaviors through interaction and reward signals. However, designing effective reward functions for RL remains challenging, especially in real-world tasks where sparse rewards are insufficient and dense rewards require elaborate design. In this work, we propose Autonomous Reinforcement learning for Complex Human-Informed Environments (ARCHIE), an unsupervised pipeline leveraging GPT-4, a pre-trained LLM, to generate reward functions directly from natural language task descriptions. The rewards are used to train RL agents in simulated environments, where we formalize the reward generation process to enhance feasibility. Additionally, GPT-4 automates the coding of task success criteria, creating a fully automated, one-shot procedure for translating human-readable text into deployable robot skills. Our approach is validated through extensive simulated experiments on single-arm and bi-manual manipulation tasks using an ABB YuMi collaborative robot, highlighting its practicality and effectiveness. Tasks are demonstrated on the real robot setup.