Bounds on the Approximation Power of Feedforward Neural Networks
This work addresses theoretical limitations in neural network design for researchers, offering incremental improvements in bounds for specific function classes.
The paper investigates the approximation power of feedforward neural networks with piecewise linear activations, establishing lower bounds on network size in terms of error, depth, and width that improve state-of-the-art bounds for classes like strongly convex functions, and provides an upper bound on differences between networks with identical weights but different activations.
The approximation power of general feedforward neural networks with piecewise linear activation functions is investigated. First, lower bounds on the size of a network are established in terms of the approximation error and network depth and width. These bounds improve upon state-of-the-art bounds for certain classes of functions, such as strongly convex functions. Second, an upper bound is established on the difference of two neural networks with identical weights but different activation functions.