LLM Collaboration With Multi-Agent Reinforcement Learning
This addresses the challenge of enabling LLMs to collaborate effectively in multi-agent systems, which is an incremental improvement over existing fine-tuning frameworks.
The paper tackles the problem of LLMs lacking coordination in multi-agent systems by modeling LLM collaboration as a cooperative Multi-Agent Reinforcement Learning problem, developing the MAGRPO algorithm which enables agents to generate high-quality responses efficiently through cooperation in writing and coding tasks.
A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using other MARL methods for LLMs and highlights the associated challenges.