On Convergence and Optimality of Best-Response Learning with Policy Types in Multiagent Systems
This work addresses coordination challenges in heterogeneous multiagent systems, offering incremental theoretical improvements for applications requiring expert-provided policy specifications.
The paper tackles the problem of coordinating actions in multiagent systems with unknown agent behaviors by analyzing convergence and optimality of best-response learning with policy types, showing that a new posterior can learn correlated distributions and providing a novel optimality characterization for expert-verified types.
While many multiagent algorithms are designed for homogeneous systems (i.e. all agents are identical), there are important applications which require an agent to coordinate its actions without knowing a priori how the other agents behave. One method to make this problem feasible is to assume that the other agents draw their latent policy (or type) from a specific set, and that a domain expert could provide a specification of this set, albeit only a partially correct one. Algorithms have been proposed by several researchers to compute posterior beliefs over such policy libraries, which can then be used to determine optimal actions. In this paper, we provide theoretical guidance on two central design parameters of this method: Firstly, it is important that the user choose a posterior which can learn the true distribution of latent types, as otherwise suboptimal actions may be chosen. We analyse convergence properties of two existing posterior formulations and propose a new posterior which can learn correlated distributions. Secondly, since the types are provided by an expert, they may be inaccurate in the sense that they do not predict the agents' observed actions. We provide a novel characterisation of optimality which allows experts to use efficient model checking algorithms to verify optimality of types.