RAM-Net: Expressive Linear Attention with Selectively Addressable Memory
This addresses the problem of balancing efficiency and expressivity in attention mechanisms for machine learning practitioners, offering a novel method that is not purely incremental.
The paper tackles the expressivity and information loss limitations of linear attention models by introducing RAM-Net, a novel architecture that uses high-dimensional sparse vectors for selective memory access, enabling exponential state scaling without extra parameters. It demonstrates superior performance in long-range retrieval tasks and competitive results in language modeling and commonsense reasoning benchmarks.
While linear attention architectures offer efficient inference, compressing unbounded history into a fixed-size memory inherently limits expressivity and causes information loss. To address this limitation, we introduce Random Access Memory Network (RAM-Net), a novel architecture designed to bridge the gap between the representational capacity of full attention and the memory efficiency of linear models. The core of RAM-Net maps inputs to high-dimensional sparse vectors serving as explicit addresses, allowing the model to selectively access a massive memory state. This design enables exponential state size scaling without additional parameters, which significantly mitigates signal interference and enhances retrieval fidelity. Moreover, the inherent sparsity ensures exceptional computational efficiency, as state updates are confined to minimal entries. Extensive experiments demonstrate that RAM-Net consistently surpasses state-of-the-art baselines in fine-grained long-range retrieval tasks and achieves competitive performance in standard language modeling and zero-shot commonsense reasoning benchmarks, validating its superior capability to capture complex dependencies with significantly reduced computational overhead.