Neural networks and rational functions
This work addresses a foundational problem in machine learning theory by establishing efficient equivalence between neural networks and rational functions, with implications for approximation theory and model interpretability, though it is incremental in extending known approximation results.
The paper tackles the problem of approximating neural networks with rational functions and vice versa, showing that ReLU networks and rational functions can approximate each other efficiently with degree or size scaling as O(polylog(1/ε)), whereas polynomials require Ω(poly(1/ε)) to approximate a single ReLU, and it demonstrates that the conversion constants depend exponentially on network layers, which is tight.
Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degree $O(\text{polylog}(1/ε))$ which is $ε$-close, and similarly for any rational function there exists a ReLU network of size $O(\text{polylog}(1/ε))$ which is $ε$-close. By contrast, polynomials need degree $Ω(\text{poly}(1/ε))$ to approximate even a single ReLU. When converting a ReLU network to a rational function as above, the hidden constants depend exponentially on the number of layers, which is shown to be tight; in other words, a compositional representation can be beneficial even for rational functions.