Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking
This provides a theoretical framework for understanding Transformers and MLP-Mixers, aiding future model design, but it is incremental as it builds on existing MetaFormers and MLP-Mixer concepts.
The paper tackled the theoretical underdevelopment of Transformer-like models by integrating hierarchical associative memory with MetaFormers to represent a Transformer block as a Hopfield network, resulting in a parallelized MLP-Mixer that matches vanilla MLP-Mixer performance after symmetry breaking, with empirical studies showing symmetry hinders image recognition tasks.
Transformers have established themselves as the leading neural network model in natural language processing and are increasingly foundational in various domains. In vision, the MLP-Mixer model has demonstrated competitive performance, suggesting that attention mechanisms might not be indispensable. Inspired by this, recent research has explored replacing attention modules with other mechanisms, including those described by MetaFormers. However, the theoretical framework for these models remains underdeveloped. This paper proposes a novel perspective by integrating Krotov's hierarchical associative memory with MetaFormers, enabling a comprehensive representation of the entire Transformer block, encompassing token-/channel-mixing modules, layer normalization, and skip connections, as a single Hopfield network. This approach yields a parallelized MLP-Mixer derived from a three-layer Hopfield network, which naturally incorporates symmetric token-/channel-mixing modules and layer normalization. Empirical studies reveal that symmetric interaction matrices in the model hinder performance in image recognition tasks. Introducing symmetry-breaking effects transitions the performance of the symmetric parallelized MLP-Mixer to that of the vanilla MLP-Mixer. This indicates that during standard training, weight matrices of the vanilla MLP-Mixer spontaneously acquire a symmetry-breaking configuration, enhancing their effectiveness. These findings offer insights into the intrinsic properties of Transformers and MLP-Mixers and their theoretical underpinnings, providing a robust framework for future model design and optimization.