The Gossiping Insert-Eliminate Algorithm for Multi-Agent Bandits
This addresses regret minimization for agents in decentralized networks, offering a significant improvement over non-collaborative approaches, though it appears incremental relative to prior multi-agent bandit work.
The paper tackles decentralized multi-agent multi-armed bandits by proposing algorithms where agents exchange arm recommendations via gossip communications, showing that with Ω(log(T)) communications, each agent's regret reduces by a factor of order N compared to no collaboration, with communication constraints having only a second-order effect.
We consider a decentralized multi-agent Multi Armed Bandit (MAB) setup consisting of $N$ agents, solving the same MAB instance to minimize individual cumulative regret. In our model, agents collaborate by exchanging messages through pairwise gossip style communications on an arbitrary connected graph. We develop two novel algorithms, where each agent only plays from a subset of all the arms. Agents use the communication medium to recommend only arm-IDs (not samples), and thus update the set of arms from which they play. We establish that, if agents communicate $Ω(\log(T))$ times through any connected pairwise gossip mechanism, then every agent's regret is a factor of order $N$ smaller compared to the case of no collaborations. Furthermore, we show that the communication constraints only have a second order effect on the regret of our algorithm. We then analyze this second order term of the regret to derive bounds on the regret-communication tradeoffs. Finally, we empirically evaluate our algorithm and conclude that the insights are fundamental and not artifacts of our bounds. We also show a lower bound which gives that the regret scaling obtained by our algorithm cannot be improved even in the absence of any communication constraints. Our results thus demonstrate that even a minimal level of collaboration among agents greatly reduces regret for all agents.