Multi-agent Off-policy Actor-Critic Reinforcement Learning for Partially Observable Environments
This addresses the challenge of decentralized multi-agent coordination without full observability, offering a model-free solution for applications like robotics or autonomous systems, though it appears incremental as it builds on existing actor-critic frameworks.
The study tackled the problem of multi-agent reinforcement learning in partially observable environments by proposing a social learning method to estimate global states within an off-policy actor-critic algorithm, achieving ε-bounded error and outperforming state-of-the-art methods in experiments.
This study proposes the use of a social learning method to estimate a global state within a multi-agent off-policy actor-critic algorithm for reinforcement learning (RL) operating in a partially observable environment. We assume that the network of agents operates in a fully-decentralized manner, possessing the capability to exchange variables with their immediate neighbors. The proposed design methodology is supported by an analysis demonstrating that the difference between final outcomes, obtained when the global state is fully observed versus estimated through the social learning method, is $\varepsilon$-bounded when an appropriate number of iterations of social learning updates are implemented. Unlike many existing dec-POMDP-based RL approaches, the proposed algorithm is suitable for model-free multi-agent reinforcement learning as it does not require knowledge of a transition model. Furthermore, experimental results illustrate the efficacy of the algorithm and demonstrate its superiority over the current state-of-the-art methods.