M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
This addresses the problem of suboptimal or misaligned behaviors in complex, coordinated MARL environments, enabling broader human participation, but it is incremental as it builds on existing human feedback and MARL techniques.
The paper tackles the challenge of designing effective reward functions in multi-agent reinforcement learning (MARL) by introducing M3HF, a framework that integrates multi-phase human feedback of mixed quality, and it significantly outperforms state-of-the-art methods in empirical results.
Designing effective reward functions in multi-agent reinforcement learning (MARL) is a significant challenge, often leading to suboptimal or misaligned behaviors in complex, coordinated environments. We introduce Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality ($\text{M}^3\text{HF}$), a novel framework that integrates multi-phase human feedback of mixed quality into the MARL training process. By involving humans with diverse expertise levels to provide iterative guidance, $\text{M}^3\text{HF}$ leverages both expert and non-expert feedback to continuously refine agents' policies. During training, we strategically pause agent learning for human evaluation, parse feedback using large language models to assign it appropriately and update reward functions through predefined templates and adaptive weights by using weight decay and performance-based adjustments. Our approach enables the integration of nuanced human insights across various levels of quality, enhancing the interpretability and robustness of multi-agent cooperation. Empirical results in challenging environments demonstrate that $\text{M}^3\text{HF}$ significantly outperforms state-of-the-art methods, effectively addressing the complexities of reward design in MARL and enabling broader human participation in the training process.