The Barron Space and the Flow-induced Function Spaces for Neural Network Models
This work addresses a foundational issue in machine learning theory for researchers, providing rigorous mathematical frameworks to control approximation and estimation errors in neural networks, though it is incremental in extending existing space concepts to new models.
The paper tackles the problem of identifying appropriate function spaces for analyzing neural network models, specifically for two-layer networks and residual neural networks, by defining the Barron space and flow-induced function space, and proving optimal direct and inverse approximation theorems along with optimal upper bounds for Rademacher complexity.
One of the key issues in the analysis of machine learning models is to identify the appropriate function space and norm for the model. This is the set of functions endowed with a quantity which can control the approximation and estimation errors by a particular machine learning model. In this paper, we address this issue for two representative neural network models: the two-layer networks and the residual neural networks. We define the Barron space and show that it is the right space for two-layer neural network models in the sense that optimal direct and inverse approximation theorems hold for functions in the Barron space. For residual neural network models, we construct the so-called flow-induced function space, and prove direct and inverse approximation theorems for this space. In addition, we show that the Rademacher complexity for bounded sets under these norms has the optimal upper bounds.