Associative Memory in Iterated Overparameterized Sigmoid Autoencoders
This work provides theoretical insights into associative memory mechanisms in neural networks, which is incremental as it builds on prior studies of overparameterized autoencoders and NTK theory.
The paper tackled the problem of implementing associative memory in overparameterized sigmoid autoencoders by analyzing their training dynamics in the infinite-width limit, finding that attractors emerge under certain conditions, with the largest Jacobian eigenvalue norm dropping below one for multiple examples as input norm increases.
Recent work showed that overparameterized autoencoders can be trained to implement associative memory via iterative maps, when the trained input-output Jacobian of the network has all of its eigenvalue norms strictly below one. Here, we theoretically analyze this phenomenon for sigmoid networks by leveraging recent developments in deep learning theory, especially the correspondence between training neural networks in the infinite-width limit and performing kernel regression with the Neural Tangent Kernel (NTK). We find that overparameterized sigmoid autoencoders can have attractors in the NTK limit for both training with a single example and multiple examples under certain conditions. In particular, for multiple training examples, we find that the norm of the largest Jacobian eigenvalue drops below one with increasing input norm, leading to associative memory.