Neural network integral representations with the ReLU activation function
This work addresses theoretical representation problems in neural networks for researchers in machine learning theory, but it appears incremental as it builds on existing integral representation concepts.
The authors derived an integral representation formula for shallow neural networks using the ReLU activation function, assuming outer weights have a finite L1-norm on the sphere, and provided a closed-form formula for univariate target functions, enabling explicit solutions for least L1-norm representations.
In this effort, we derive a formula for the integral representation of a shallow neural network with the ReLU activation function. We assume that the outer weighs admit a finite $L_1$-norm with respect to Lebesgue measure on the sphere. For univariate target functions we further provide a closed-form formula for all possible representations. Additionally, in this case our formula allows one to explicitly solve the least $L_1$-norm neural network representation for a given function.