Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning
This addresses the challenge of manual reward engineering for cooperative multi-agent reinforcement learning, though it appears incremental as it builds on existing reward design methods with LLM automation.
The researchers tackled the problem of designing effective auxiliary rewards for cooperative multi-agent systems by introducing an automated framework that uses large language models to synthesize reward programs from environment instrumentation, achieving superior task returns and delivery counts across four Overcooked-AI layouts, with the most pronounced gains in environments with interaction bottlenecks.
Designing effective auxiliary rewards for cooperative multi-agent systems remains a precarious task; misaligned incentives risk inducing suboptimal coordination, especially where sparse task feedback fails to provide sufficient grounding. This study introduces an automated reward design framework that leverages large language models to synthesize executable reward programs from environment instrumentation. The procedure constrains candidate programs within a formal validity envelope and evaluates their efficacy by training policies from scratch under a fixed computational budget; selection depends exclusively on the sparse task return. The framework is evaluated across four distinct Overcooked-AI layouts characterized by varied corridor congestion, handoff dependencies, and structural asymmetries. Iterative search generations consistently yield superior task returns and delivery counts, with the most pronounced gains occurring in environments dominated by interaction bottlenecks. Diagnostic analysis of the synthesized shaping components indicates increased interdependence in action selection and improved signal alignment in coordination-intensive tasks. These results demonstrate that the search for objectivegrounded reward programs can mitigate the burden of manual engineering while producing shaping signals compatible with cooperative learning under finite budgets.