Learning Empirical Evidence Equilibria under Weak Environmental Coupling
Provides theoretical guarantees for equilibrium emergence in decentralized multi-agent systems with bounded rationality, addressing a key challenge in multi-agent reinforcement learning.
This work proves that an Empirical Evidence Equilibrium (EEE) emerges from joint Q-value iteration dynamics in multi-agent systems when the coupling between agents' actions and the environment is sufficiently weak, and extends this result to softmax policies.
Strategic multi-agent systems are fundamentally characterized by decentralization, uncertainty, and ambiguity. Agents operating under limited observations will often need to make decisions based on simplified internal models of the environment, reflecting bounded rationality in both computational capacity and environmental knowledge. The Empirical Evidence Equilibrium (EEE) framework explicitly accounts for these limitations by modeling each agent as forming a potentially misspecified belief derived from signals obtained through partial observations of the environment. The resulting equilibrium concept captures the system's steady state under bounded rationality and decentralization. In this work, we study games in which the environment dynamics are driven jointly by exogenous factors and agents' actions. We analyze agent behavior under Q-value iteration where each agent independently forms a belief model, computes Q-values, and derives a greedy strategy, yet the collective actions of all agents jointly shape the environment each agent faces at the next stage. We prove that despite this decentralization, an EEE emerges from the joint dynamics when the coupling between agents' actions and the environment is sufficiently weak. We further extend this result to softmax policies, establishing a contraction result under a sufficient coupling condition.