Understanding neural networks with reproducing kernel Banach spaces
This work provides theoretical insights into neural network properties, but it is incremental as it simplifies and extends prior results.
The paper tackles the challenge of characterizing function spaces for neural networks by applying reproducing kernel Banach spaces theory, proving a representer theorem for such spaces that includes one hidden layer networks and showing norm characterization via inverse Radon transforms for ReLU activities.
Characterizing the function spaces corresponding to neural networks can provide a way to understand their properties. In this paper we discuss how the theory of reproducing kernel Banach spaces can be used to tackle this challenge. In particular, we prove a representer theorem for a wide class of reproducing kernel Banach spaces that admit a suitable integral representation and include one hidden layer neural networks of possibly infinite width. Further, we show that, for a suitable class of ReLU activation functions, the norm in the corresponding reproducing kernel Banach space can be characterized in terms of the inverse Radon transform of a bounded real measure, with norm given by the total variation norm of the measure. Our analysis simplifies and extends recent results in [34,29,30].