Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition
This addresses the challenge of emergent phenomena in neural networks for researchers, providing insights into grokking as a phase transition, but it is incremental as it builds on existing studies of grokking.
The paper tackles the problem of understanding grokking in neural networks, where models suddenly generalize after memorization, by using higher-order mutual information to analyze neuron interactions and identifying distinct training phases that allow anticipation of grokking, attributing it to an emergent phase transition.
This paper studies emergent phenomena in neural networks by focusing on grokking where models suddenly generalize after delayed memorization. To understand this phase transition, we utilize higher-order mutual information to analyze the collective behavior (synergy) and shared properties (redundancy) between neurons during training. We identify distinct phases before grokking allowing us to anticipate when it occurs. We attribute grokking to an emergent phase transition caused by the synergistic interactions between neurons as a whole. We show that weight decay and weight initialization can enhance the emergent phase.