iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixer
This work provides a theoretical principle for designing and analyzing Transformer-like models, though it is incremental as it builds on existing connections between Hopfield networks and MLP-Mixer.
The authors tackled the problem of understanding and generalizing Transformer-like architectures by linking hierarchical Hopfield networks to MLP-Mixer, resulting in iMixer, which achieves performance comparable to or better than baseline MLP-Mixer on image classification tasks.
In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models that compete with Transformers, such as the MLP-Mixer. Despite their weak inductive bias, these models have achieved performance comparable to well-studied convolutional neural networks. Recent studies on modern Hopfield networks suggest the correspondence between certain energy-based associative memory models and Transformers or MLP-Mixer, and shed some light on the theoretical background of the Transformer-type architectures design. In this paper, we generalize the correspondence to the recently introduced hierarchical Hopfield network, and find iMixer, a novel generalization of MLP-Mixer model. Unlike ordinary feedforward neural networks, iMixer involves MLP layers that propagate forward from the output side to the input side. We characterize the module as an example of invertible, implicit, and iterative mixing module. We evaluate the model performance with various datasets on image classification tasks, and find that iMixer, despite its unique architecture, exhibits stable learning capabilities and achieves performance comparable to or better than the baseline vanilla MLP-Mixer. The results imply that the correspondence between the Hopfield networks and the Mixer models serves as a principle for understanding a broader class of Transformer-like architecture designs.