FireAct: Toward Language Agent Fine-tuning
This work addresses the challenge of enhancing language agents for tasks like question answering, though it is incremental as it builds on existing fine-tuning methods applied to a new context.
The paper tackles the problem of improving language agents by fine-tuning their backbone language models, rather than relying on few-shot prompting, and demonstrates that fine-tuning with agent trajectories leads to significant performance gains, such as a 77% increase on HotpotQA with Llama2-7B.
Recent efforts have augmented language models (LMs) with external tools or environments, leading to the development of language agents that can reason and act. However, most of these agents rely on few-shot prompting techniques with off-the-shelf LMs. In this paper, we investigate and argue for the overlooked direction of fine-tuning LMs to obtain language agents. Using a setup of question answering (QA) with a Google search API, we explore a variety of base LMs, prompting methods, fine-tuning data, and QA tasks, and find language agents are consistently improved after fine-tuning their backbone LMs. For example, fine-tuning Llama2-7B with 500 agent trajectories generated by GPT-4 leads to a 77% HotpotQA performance increase. Furthermore, we propose FireAct, a novel approach to fine-tuning LMs with trajectories from multiple tasks and prompting methods, and show having more diverse fine-tuning data can further improve agents. Along with other findings regarding scaling effects, robustness, generalization, efficiency and cost, our work establishes comprehensive benefits of fine-tuning LMs for agents, and provides an initial set of experimental designs, insights, as well as open questions toward language agent fine-tuning.