Game On: Towards Language Models as RL Experimenters
This work addresses the challenge of reducing human effort in reinforcement learning workflows for robotics, though it appears incremental as it builds on existing VLM and RL methods.
The authors tackled the problem of automating reinforcement learning experiments for embodied agents by proposing an agent architecture that uses a Vision-Language Model (VLM) to monitor progress, propose tasks, decompose them into skills, and retrieve skills, enabling automated curricula. They demonstrated a prototype using a standard Gemini model to steer data collection, showing that the collected data aids in learning and improving control policies in robotics, with promising results for skill library growth and training progress judgment.
We propose an agent architecture that automates parts of the common reinforcement learning experiment workflow, to enable automated mastery of control domains for embodied agents. To do so, it leverages a VLM to perform some of the capabilities normally required of a human experimenter, including the monitoring and analysis of experiment progress, the proposition of new tasks based on past successes and failures of the agent, decomposing tasks into a sequence of subtasks (skills), and retrieval of the skill to execute - enabling our system to build automated curricula for learning. We believe this is one of the first proposals for a system that leverages a VLM throughout the full experiment cycle of reinforcement learning. We provide a first prototype of this system, and examine the feasibility of current models and techniques for the desired level of automation. For this, we use a standard Gemini model, without additional fine-tuning, to provide a curriculum of skills to a language-conditioned Actor-Critic algorithm, in order to steer data collection so as to aid learning new skills. Data collected in this way is shown to be useful for learning and iteratively improving control policies in a robotics domain. Additional examination of the ability of the system to build a growing library of skills, and to judge the progress of the training of those skills, also shows promising results, suggesting that the proposed architecture provides a potential recipe for fully automated mastery of tasks and domains for embodied agents.