Towards an Algebraic Framework For Approximating Functions Using Neural Network Polynomials
This work provides a theoretical framework for understanding neural network approximations, which is incremental as it extends existing calculus.
The paper tackles the problem of approximating functions using neural networks with algebraic structures, showing that neural network polynomials, exponentials, sine, and cosines can approximate their real counterparts with polynomial growth in parameters and depth relative to accuracy.
We make the case for neural network objects and extend an already existing neural network calculus explained in detail in Chapter 2 on \cite{bigbook}. Our aim will be to show that, yes, indeed, it makes sense to talk about neural network polynomials, neural network exponentials, sine, and cosines in the sense that they do indeed approximate their real number counterparts subject to limitations on certain of their parameters, $q$, and $\varepsilon$. While doing this, we show that the parameter and depth growth are only polynomial on their desired accuracy (defined as a 1-norm difference over $\mathbb{R}$), thereby showing that this approach to approximating, where a neural network in some sense has the structural properties of the function it is approximating is not entire intractable.