Online Planning for Decentralized Stochastic Control with Partial History Sharing
This work addresses computational challenges in decentralized decision-making for applications like network defense, though it is incremental as it generalizes existing heuristic solvers.
The paper tackles the intractability of decentralized stochastic control by leveraging partial history sharing to reduce the policy search space, proposing an online tree-search algorithm that is provably convergent and model-free, with numerical results demonstrating its performance in a collaborative intrusion response scenario.
In decentralized stochastic control, standard approaches for sequential decision-making, e.g. dynamic programming, quickly become intractable due to the need to maintain a complex information state. Computational challenges are further compounded if agents do not possess complete model knowledge. In this paper, we take advantage of the fact that in many problems agents share some common information, or history, termed partial history sharing. Under this information structure the policy search space is greatly reduced. We propose a provably convergent, online tree-search based algorithm that does not require a closed-form model or explicit communication among agents. Interestingly, our algorithm can be viewed as a generalization of several existing heuristic solvers for decentralized partially observable Markov decision processes. To demonstrate the applicability of the model, we propose a novel collaborative intrusion response model, where multiple agents (defenders) possessing asymmetric information aim to collaboratively defend a computer network. Numerical results demonstrate the performance of our algorithm.