Why Shallow Networks Struggle to Approximate and Learn High Frequencies
This addresses a fundamental computational bottleneck in neural network theory, with implications for designing more efficient and stable models, though it is incremental in building on existing analysis.
The paper tackles the problem of why shallow neural networks struggle with high frequencies in approximation and learning, revealing that finite precision, computational cost, and instability limit their performance, with explicit mathematical and numerical evidence provided.
In this work, we present a comprehensive study combining mathematical and computational analysis to explain why a two-layer neural network struggles to handle high frequencies in both approximation and learning, especially when machine precision, numerical noise, and computational cost are significant factors in practice. Specifically, we investigate the following fundamental computational issues: (1) the minimal numerical error achievable under finite precision, (2) the computational cost required to attain a given accuracy, and (3) the stability of the method with respect to perturbations. The core of our analysis lies in the conditioning of the representation and its learning dynamics. Explicit answers to these questions are provided, along with supporting numerical evidence.