Exploring Grokking: Experimental and Mechanistic Investigations
This addresses a fundamental problem in understanding neural network generalization for researchers in machine learning, but it is incremental as it builds on existing viewpoints without introducing new methods.
The paper investigates the grokking phenomenon in over-parameterized neural networks, where networks initially memorize training data with poor generalization but later transition sharply to perfect generalization after prolonged training, exploring experimental insights into factors like training data fraction and optimization.
The phenomenon of grokking in over-parameterized neural networks has garnered significant interest. It involves the neural network initially memorizing the training set with zero training error and near-random test error. Subsequent prolonged training leads to a sharp transition from no generalization to perfect generalization. Our study comprises extensive experiments and an exploration of the research behind the mechanism of grokking. Through experiments, we gained insights into its behavior concerning the training data fraction, the model, and the optimization. The mechanism of grokking has been a subject of various viewpoints proposed by researchers, and we introduce some of these perspectives.