Meta-RL Induces Exploration in Language Agents
This addresses the issue of inefficient exploration and adaptation in language agents for interactive tasks, representing a novel method rather than an incremental improvement.
The paper tackles the problem of RL-trained language agents struggling with active exploration and adaptation in multi-turn tasks by introducing LaMer, a Meta-RL framework that enables agents to explore and learn from environment feedback at test time, resulting in performance gains of 11%, 14%, and 19% on Sokoban, MineSweeper, and Webshop, respectively.
Reinforcement learning (RL) has enabled the training of large language model (LLM) agents to interact with the environment and to solve multi-turn long-horizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LaMer, a general Meta-RL framework that enables LLM agents to actively explore and learn from the environment feedback at test time. LaMer consists of two key components: (i) a cross-episode training framework to encourage exploration and long-term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environments show that LaMer significantly improves performance over RL baselines, with 11%, 14%, and 19% performance gains on Sokoban, MineSweeper and Webshop, respectively. Moreover, LaMer also demonstrates better generalization to more challenging or previously unseen tasks compared to the RL-trained agents. Overall, our results demonstrate that Meta-RL provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.