Provable Benefits of Sinusoidal Activation for Modular Addition
This work addresses a foundational problem in neural network expressivity and generalization for researchers in machine learning theory, though it is incremental as it builds on prior studies of activation functions.
The paper tackles the problem of learning modular addition with two-layer neural networks by analyzing the role of activation functions, showing that sinusoidal activations enable exact realizations with constant width and nearly optimal sample complexity, while ReLU networks require linearly scaling width and fail in certain cases.
This paper studies the role of activation functions in learning modular addition with two-layer neural networks. We first establish a sharp expressivity gap: sine MLPs admit width-$2$ exact realizations for any fixed length $m$ and, with bias, width-$2$ exact realizations uniformly over all lengths. In contrast, the width of ReLU networks must scale linearly with $m$ to interpolate, and they cannot simultaneously fit two lengths with different residues modulo $p$. We then provide a novel Natarajan-dimension generalization bound for sine networks, yielding nearly optimal sample complexity $\widetilde{\mathcal{O}}(p)$ for ERM over constant-width sine networks. We also derive width-independent, margin-based generalization for sine networks in the overparametrized regime and validate it. Empirically, sine networks generalize consistently better than ReLU networks across regimes and exhibit strong length extrapolation.