AGILE: A Novel Reinforcement Learning Framework of LLM Agents
This addresses the challenge of creating more capable and interactive AI agents for applications like question answering, though it appears incremental as it builds on existing RL and LLM methods.
The paper tackles the problem of building LLM agents for complex conversational tasks by introducing AGILE, a reinforcement learning framework that integrates memory, tools, expert consultation, and reflection, and shows that it outperforms GPT-4 agents on datasets like ProductQA, MedMCQA, and HotPotQA using 7B and 13B LLMs.
We introduce a novel reinforcement learning framework of LLM agents named AGILE (AGent that Interacts and Learns from Environments) designed to perform complex conversational tasks with users, leveraging LLMs, memory, tools, and interactions with experts. The agent possesses capabilities beyond conversation, including reflection, tool usage, and expert consultation. We formulate the construction of such an LLM agent as a reinforcement learning (RL) problem, in which the LLM serves as the policy model. We fine-tune the LLM using labeled data of actions and the PPO algorithm. We focus on question answering and release a dataset for agents called ProductQA, comprising challenging questions in online shopping. Our extensive experiments on ProductQA, MedMCQA and HotPotQA show that AGILE agents based on 7B and 13B LLMs trained with PPO can outperform GPT-4 agents. Our ablation study highlights the indispensability of memory, tools, consultation, reflection, and reinforcement learning in achieving the agent's strong performance. Datasets and code are available at https://github.com/bytarnish/AGILE.