In-context denoising with one-layer transformers: connections between attention and associative memory retrieval
This work solidifies the link between associative memory and attention mechanisms, relevant for understanding in-context learning in AI, but it is incremental as it builds on prior connections identified by others.
The paper tackles the connection between attention mechanisms and associative memory by introducing an in-context denoising task, showing theoretically and empirically that a single-layer transformer can solve certain denoising problems optimally, with the attention layer performing a gradient descent update that yields better solutions than standard retrieval methods.
We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfield networks. Using a Bayesian framework, we show theoretically and empirically that certain restricted denoising problems can be solved optimally even by a single-layer transformer. We demonstrate that a trained attention layer processes each denoising prompt by performing a single gradient descent update on a context-aware DAM energy landscape, where context tokens serve as associative memories and the query token acts as an initial state. This one-step update yields better solutions than exact retrieval of either a context token or a spurious local minimum, providing a concrete example of DAM networks extending beyond the standard retrieval paradigm. Overall, this work solidifies the link between associative memory and attention mechanisms first identified by Ramsauer et al., and demonstrates the relevance of associative memory models in the study of in-context learning.