LangMARL: Natural Language Multi-Agent Reinforcement Learning
This addresses the multi-agent credit assignment problem for LLM-based systems, offering a novel integration of MARL techniques to enhance coordination in dynamic environments, though it builds on existing MARL concepts.
The paper tackles the problem of LLM agents struggling to evolve coordination strategies in dynamic environments due to coarse global outcomes obscuring causal signals, proposing LangMARL to incorporate credit assignment and policy gradient evolution from MARL into language space, resulting in improved sample efficiency, interpretability, and strong generalization in cooperative multi-agent tasks.
Large language model (LLM) agents struggle to autonomously evolve coordination strategies in dynamic environments, largely because coarse global outcomes obscure the causal signals needed for local policy refinement. We identify this bottleneck as a multi-agent credit assignment problem, which has long been studied in classical multi-agent reinforcement learning (MARL) but remains underaddressed in LLM-based systems. Building on this observation, we propose LangMARL, a framework that brings credit assignment and policy gradient evolution from cooperative MARL into the language space. LangMARL introduces agent-level language credit assignment, pioneers gradient evolution in language space for policy improvement, and summarizes task-relevant causal relations from replayed trajectories to provide dense feedback and improve convergence under sparse rewards. Extensive experiments across diverse cooperative multi-agent tasks demonstrate improved sample efficiency, interpretability, and strong generalization.