Distributed Associative Memory via Online Convex Optimization
This addresses distributed associative memory for multi-agent systems, representing an incremental advance with a novel method for a known bottleneck.
The paper tackles the problem of distributed associative memory where agents maintain local memories and selectively share information, introducing a distributed online gradient descent method over routing trees. The result shows sublinear regret guarantees theoretically and consistent outperformance over existing online optimization baselines experimentally.
An associative memory (AM) enables cue-response recall, and associative memorization has recently been noted to underlie the operation of modern neural architectures such as Transformers. This work addresses a distributed setting where agents maintain a local AM to recall their own associations as well as selective information from others. Specifically, we introduce a distributed online gradient descent method that optimizes local AMs at different agents through communication over routing trees. Our theoretical analysis establishes sublinear regret guarantees, and experiments demonstrate that the proposed protocol consistently outperforms existing online optimization baselines.