Grokking Modular Polynomials
This work addresses the generalization limitations of neural networks on modular arithmetic, which is an incremental advance in understanding network learning dynamics for specific mathematical tasks.
The authors tackled the problem of neural networks failing to generalize on modular arithmetic tasks by extending analytical solutions to include modular multiplication and multi-term addition, and combining these to handle arbitrary modular polynomials, achieving generalization (grokking) in trained networks.
Neural networks readily learn a subset of the modular arithmetic tasks, while failing to generalize on the rest. This limitation remains unmoved by the choice of architecture and training strategies. On the other hand, an analytical solution for the weights of Multi-layer Perceptron (MLP) networks that generalize on the modular addition task is known in the literature. In this work, we (i) extend the class of analytical solutions to include modular multiplication as well as modular addition with many terms. Additionally, we show that real networks trained on these datasets learn similar solutions upon generalization (grokking). (ii) We combine these "expert" solutions to construct networks that generalize on arbitrary modular polynomials. (iii) We hypothesize a classification of modular polynomials into learnable and non-learnable via neural networks training; and provide experimental evidence supporting our claims.