A function space analysis of finite neural networks with insights from sampling theory
This work offers theoretical insights into neural network behavior for researchers in machine learning, though it is incremental as it extends prior analyses from infinite to finite cases.
The paper tackles the analysis of function spaces represented by finite neural networks using sampling theory, showing that under a finite input domain, multi-layer networks with non-expansive activations produce smooth functions, and providing novel error bounds for univariate networks under band-limited input assumptions.
This work suggests using sampling theory to analyze the function space represented by neural networks. First, it shows, under the assumption of a finite input domain, which is the common case in training neural networks, that the function space generated by multi-layer networks with non-expansive activation functions is smooth. This extends over previous works that show results for the case of infinite width ReLU networks. Then, under the assumption that the input is band-limited, we provide novel error bounds for univariate neural networks. We analyze both deterministic uniform and random sampling showing the advantage of the former.