A Roadmap Towards Improving Multi-Agent Reinforcement Learning With Causal Discovery And Inference
This work addresses the largely unexplored problem of integrating causal reasoning into MARL for researchers, but it is incremental as it focuses on initial exploration and outlines future research directions.
The paper investigates applying causal reasoning to Multi-Agent Reinforcement Learning (MARL) to enhance policy efficacy, convergence efficiency, generalization, safety, and interpretability, finding mixed results from testing a simple causal augmentation in state-of-the-art MARL scenarios.
Causal reasoning is increasingly used in Reinforcement Learning (RL) to improve the learning process in several dimensions: efficacy of learned policies, efficiency of convergence, generalisation capabilities, safety and interpretability of behaviour. However, applications of causal reasoning to Multi-Agent RL (MARL) are still mostly unexplored. In this paper, we take the first step in investigating the opportunities and challenges of applying causal reasoning in MARL. We measure the impact of a simple form of causal augmentation in state-of-the-art MARL scenarios increasingly requiring cooperation, and with state-of-the-art MARL algorithms exploiting various degrees of collaboration between agents. Then, we discuss the positive as well as negative results achieved, giving us the chance to outline the areas where further research may help to successfully transfer causal RL to the multi-agent setting.