The phase diagram of approximation rates for deep neural networks
This provides foundational theoretical insights for machine learning researchers, clarifying the limits and capabilities of neural network approximation, though it is incremental in extending prior results to broader classes.
The paper tackles the problem of understanding approximation rates for deep neural networks across different activation functions and architectures, proving that piecewise polynomial activations share the same phase diagram, fixed-width networks can adapt to smoothness for near-optimal rates, and periodic activations enable nearly exponential rates via lookup operations.
We explore the phase diagram of approximation rates for deep neural networks and prove several new theoretical results. In particular, we generalize the existing result on the existence of deep discontinuous phase in ReLU networks to functional classes of arbitrary positive smoothness, and identify the boundary between the feasible and infeasible rates. Moreover, we show that all networks with a piecewise polynomial activation function have the same phase diagram. Next, we demonstrate that standard fully-connected architectures with a fixed width independent of smoothness can adapt to smoothness and achieve almost optimal rates. Finally, we consider deep networks with periodic activations ("deep Fourier expansion") and prove that they have very fast, nearly exponential approximation rates, thanks to the emerging capability of the network to implement efficient lookup operations.