On the expressivity of deep Heaviside networks
This addresses theoretical limitations in neural network expressivity for researchers in machine learning theory, but it is incremental as it builds on existing network architectures.
The paper tackled the limited expressiveness of deep Heaviside networks by showing that skip connections or linear activations can overcome this, providing bounds for VC dimensions and approximation rates, and deriving statistical convergence rates for nonparametric regression.
We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We provide lower and upper bounds for the Vapnik-Chervonenkis (VC) dimensions and approximation rates of these network classes. As an application, we derive statistical convergence rates for DHN fits in the nonparametric regression model.