Multi-Agent Reinforcement Learning with Multi-Step Generative Models
This work addresses a domain-specific problem for robots interacting in shared workspaces, offering an incremental improvement over existing methods.
The paper tackles the problem of model-based reinforcement learning in multi-agent continuous control by proposing disentangled variational auto-encoder models to capture agent interactions, demonstrating improved sample efficiency over model-free baselines and the ability to learn both cooperative and adversarial behaviors from the same data.
We consider model-based reinforcement learning (MBRL) in 2-agent, high-fidelity continuous control problems -- an important domain for robots interacting with other agents in the same workspace. For non-trivial dynamical systems, MBRL typically suffers from accumulating errors. Several recent studies have addressed this problem by learning latent variable models for trajectory segments and optimizing over behavior in the latent space. In this work, we investigate whether this approach can be extended to 2-agent competitive and cooperative settings. The fundamental challenge is how to learn models that capture interactions between agents, yet are disentangled to allow for optimization of each agent behavior separately. We propose such models based on a disentangled variational auto-encoder, and demonstrate our approach on a simulated 2-robot manipulation task, where one robot can either help or distract the other. We show that our approach has better sample efficiency than a strong model-free RL baseline, and can learn both cooperative and adversarial behavior from the same data.