Influence-Augmented Online Planning for Complex Environments
This addresses the challenge of efficient online planning for agents in complex, multi-agent environments where slow simulators limit performance, offering a domain-specific improvement.
The paper tackles the problem of real-time planning in complex environments with computationally demanding simulators by proposing influence-augmented online planning, which transforms a full simulator into a faster local one using machine learning to capture relevant influences, resulting in higher real-time planning performance compared to using the full simulator.
How can we plan efficiently in real time to control an agent in a complex environment that may involve many other agents? While existing sample-based planners have enjoyed empirical success in large POMDPs, their performance heavily relies on a fast simulator. However, real-world scenarios are complex in nature and their simulators are often computationally demanding, which severely limits the performance of online planners. In this work, we propose influence-augmented online planning, a principled method to transform a factored simulator of the entire environment into a local simulator that samples only the state variables that are most relevant to the observation and reward of the planning agent and captures the incoming influence from the rest of the environment using machine learning methods. Our main experimental results show that planning on this less accurate but much faster local simulator with POMCP leads to higher real-time planning performance than planning on the simulator that models the entire environment.