Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
This work addresses the need for theoretical complexity measures in neural networks, but it appears incremental as it builds on existing path-based norm methods.
The paper tackles the problem of deriving complexity controls for neural networks with general activation functions by approximating them with one-dimensional ReLU networks, resulting in path-based norms for two-layer and deep residual networks.
A simple approach is proposed to obtain complexity controls for neural networks with general activation functions. The approach is motivated by approximating the general activation functions with one-dimensional ReLU networks, which reduces the problem to the complexity controls of ReLU networks. Specifically, we consider two-layer networks and deep residual networks, for which path-based norms are derived to control complexities. We also provide preliminary analyses of the function spaces induced by these norms and a priori estimates of the corresponding regularized estimators.