The Reciprocity Gradient
This work addresses the fundamental optimization difficulty in multi-agent strategic interactions for learning agents, providing a method that avoids the collapse of sample-based approaches.
The paper identifies the influence attribution problem in multi-agent communication and introduces the reciprocity gradient, which backpropagates reward gradients through opponent policy estimators to jointly optimize actions and signals. Empirically, the method recovers near-optimal context-sensitive policies while sample-based baselines collapse into constant-output policies.
Communication is fundamental to sustaining reciprocity and cooperation in strategic interactions. We identify and formulate the influence attribution problem as the central optimization difficulty inherent in such dynamics for a learning agent: any action or signal the agent emits reshapes the reputations of many third parties along combinatorially branching paths before feeding back into its own future rewards, forcing the agent to account for all of these indirect channels at once when choosing every action. To address this, we introduce the reciprocity gradient, which explicitly backpropagates reward gradients through private estimators of opponents' policies trained from public observations. The gradient flows through the reputation chain itself analytically, rather than being estimated from sampled returns. It jointly optimizes actions and evaluative signals without intrinsic rewards or reward shaping. Empirically, the method recovers near-optimal context-sensitive policies, while sample-based baselines collapse into constant-output policies.