Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits
This provides insights into generalization mechanisms in neural networks for problems involving symmetries, though it is incremental in focusing on classical Hopfield networks.
The paper tackled the problem of how Hopfield networks can learn graph isomorphism classes from small random samples, showing they can infer the full isomorphism class efficiently with a polynomial sample complexity bound.
Many learning problems involve symmetries, and while invariance can be built into neural architectures, it can also emerge implicitly when training on group-structured data. We study this phenomenon in classical Hopfield networks and show they can infer the full isomorphism class of a graph from a small random sample. Our results reveal that: (i) graph isomorphism classes can be represented within a three-dimensional invariant subspace, (ii) using gradient descent to minimize energy flow (MEF) has an implicit bias toward norm-efficient solutions, which underpins a polynomial sample complexity bound for learning isomorphism classes, and (iii) across multiple learning rules, parameters converge toward the invariant subspace as sample sizes grow. Together, these findings highlight a unifying mechanism for generalization in Hopfield networks: a bias toward norm efficiency in learning drives the emergence of approximate invariance under group-structured data.