Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning
This work addresses the problem of offline multi-agent reinforcement learning for risk-sensitive applications like drone networks, offering a novel method that is incremental in building upon existing techniques.
The paper tackles the challenge of applying reinforcement learning to multi-agent systems when only offline data is available, proposing a scheme that integrates distributional RL and conservative Q-learning to handle uncertainties, and demonstrates its advantages in drone network trajectory planning with concrete performance improvements.
Reinforcement learning (RL) has been widely adopted for controlling and optimizing complex engineering systems such as next-generation wireless networks. An important challenge in adopting RL is the need for direct access to the physical environment. This limitation is particularly severe in multi-agent systems, for which conventional multi-agent reinforcement learning (MARL) requires a large number of coordinated online interactions with the environment during training. When only offline data is available, a direct application of online MARL schemes would generally fail due to the epistemic uncertainty entailed by the lack of exploration during training. In this work, we propose an offline MARL scheme that integrates distributional RL and conservative Q-learning to address the environment's inherent aleatoric uncertainty and the epistemic uncertainty arising from the use of offline data. We explore both independent and joint learning strategies. The proposed MARL scheme, referred to as multi-agent conservative quantile regression, addresses general risk-sensitive design criteria and is applied to the trajectory planning problem in drone networks, showcasing its advantages.