DYSTIL: Dynamic Strategy Induction with Large Language Models for Reinforcement Learning
This addresses challenges in reinforcement learning for AI agents, offering improved performance and interpretability, though it is incremental as it builds on existing LLM and RL techniques.
The paper tackled the problem of poor generalization and low sample efficiency in reinforcement learning from expert demonstrations by proposing DYSTIL, a framework that integrates large language models to dynamically generate textual strategies, resulting in a 17.75% higher average success rate compared to state-of-the-art methods.
Reinforcement learning from expert demonstrations has long remained a challenging research problem, and existing state-of-the-art methods using behavioral cloning plus further RL training often suffer from poor generalization, low sample efficiency, and poor model interpretability. Inspired by the strong reasoning abilities of large language models (LLMs), we propose a novel strategy-based reinforcement learning framework integrated with LLMs called DYnamic STrategy Induction with Llms for reinforcement learning (DYSTIL) to overcome these limitations. DYSTIL dynamically queries a strategy-generating LLM to induce textual strategies based on advantage estimations and expert demonstrations, and gradually internalizes induced strategies into the RL agent through policy optimization to improve its performance through boosting policy generalization and enhancing sample efficiency. It also provides a direct textual channel to observe and interpret the evolution of the policy's underlying strategies during training. We test DYSTIL over challenging RL environments from Minigrid and BabyAI, and empirically demonstrate that DYSTIL significantly outperforms state-of-the-art baseline methods by 17.75% in average success rate while also enjoying higher sample efficiency during the learning process.