Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok
This addresses the time-consuming hyper-parameter search for grokking, benefiting researchers in machine learning optimization, though it is incremental as it builds on existing observations of grokking phenomena.
The paper tackles the problem of predicting grokking in neural networks, where perfect generalization emerges late after overfitting, by proposing a low-cost method that analyzes early training epochs to forecast its occurrence, achieving prediction without extensive training.
This paper focuses on predicting the occurrence of grokking in neural networks, a phenomenon in which perfect generalization emerges long after signs of overfitting or memorization are observed. It has been reported that grokking can only be observed with certain hyper-parameters. This makes it critical to identify the parameters that lead to grokking. However, since grokking occurs after a large number of epochs, searching for the hyper-parameters that lead to it is time-consuming. In this paper, we propose a low-cost method to predict grokking without training for a large number of epochs. In essence, by studying the learning curve of the first few epochs, we show that one can predict whether grokking will occur later on. Specifically, if certain oscillations occur in the early epochs, one can expect grokking to occur if the model is trained for a much longer period of time. We propose using the spectral signature of a learning curve derived by applying the Fourier transform to quantify the amplitude of low-frequency components to detect the presence of such oscillations. We also present additional experiments aimed at explaining the cause of these oscillations and characterizing the loss landscape.