Optimal Coordinated Planning Amongst Self-Interested Agents with Private State
This addresses coordination challenges in multi-agent systems like robotics or economics, but it is incremental as it builds on existing MDP and game theory frameworks.
The paper tackles the problem of coordinating self-interested agents with private state in a multi-agent system, proposing an incentive-compatible mechanism that achieves the optimal joint plan in a Markov perfect equilibrium, with an efficient factored algorithm using Gittins indices for a special case.
Consider a multi-agent system in a dynamic and uncertain environment. Each agent's local decision problem is modeled as a Markov decision process (MDP) and agents must coordinate on a joint action in each period, which provides a reward to each agent and causes local state transitions. A social planner knows the model of every agent's MDP and wants to implement the optimal joint policy, but agents are self-interested and have private local state. We provide an incentive-compatible mechanism for eliciting state information that achieves the optimal joint plan in a Markov perfect equilibrium of the induced stochastic game. In the special case in which local problems are Markov chains and agents compete to take a single action in each period, we leverage Gittins allocation indices to provide an efficient factored algorithm and distribute computation of the optimal policy among the agents. Distributed, optimal coordinated learning in a multi-agent variant of the multi-armed bandit problem is obtained as a special case.