A remark on a paper of Krotov and Hopfield [arXiv:2008.06996]
This work connects recent machine learning models to foundational neurobiological theories, but it is incremental as it primarily extends existing theoretical insights without introducing new methods or data.
The paper demonstrates that the layers of the MLP-mixer model and an equivalent model can be recovered from a biologically plausible microscopic theory for associative memory, as proposed by Krotov and Hopfield, linking these modern machine learning architectures to established neurobiological frameworks.
In their recent paper titled "Large Associative Memory Problem in Neurobiology and Machine Learning" [arXiv:2008.06996] the authors gave a biologically plausible microscopic theory from which one can recover many dense associative memory models discussed in the literature. We show that the layers of the recent "MLP-mixer" [arXiv:2105.01601] as well as the essentially equivalent model in [arXiv:2105.02723] are amongst them.