Training Agents with Weakly Supervised Feedback from Large Language Models
This addresses the limitation of existing methods that require expert data or specific feedback, enabling broader application of LLM agents in complex tasks.
The paper tackles the problem of training LLM-based agents without expert trajectories or definitive environmental feedback by using weakly supervised signals from a critic LLM, resulting in agents that show consistent improvement and achieve performance comparable to GPT-4 on the API-bank dataset despite using smaller open-source models.
Large Language Models (LLMs) offer a promising basis for creating agents that can tackle complex tasks through iterative environmental interaction. Existing methods either require these agents to mimic expert-provided trajectories or rely on definitive environmental feedback for reinforcement learning which limits their application to specific scenarios like gaming or code generation. This paper introduces a novel training method for LLM-based agents using weakly supervised signals from a critic LLM, bypassing the need for expert trajectories or definitive feedback. Our agents are trained in iterative manner, where they initially generate trajectories through environmental interaction. Subsequently, a critic LLM selects a subset of good trajectories, which are then used to update the agents, enabling them to generate improved trajectories in the next iteration. Extensive tests on the API-bank dataset show consistent improvement in our agents' capabilities and comparable performance to GPT-4, despite using open-source models with much fewer parameters.