Nonparametric regression with modified ReLU networks
This work addresses regression problems for researchers in nonparametric statistics and machine learning, but it appears incremental as it modifies existing ReLU networks rather than introducing a new paradigm.
The authors tackled regression estimation using modified ReLU neural networks with weight matrices altered by a function α, achieving empirical risk minimizers that attain, up to a logarithmic factor, the minimax rate of prediction for unknown β-smooth functions.
We consider regression estimation with modified ReLU neural networks in which network weight matrices are first modified by a function $α$ before being multiplied by input vectors. We give an example of continuous, piecewise linear function $α$ for which the empirical risk minimizers over the classes of modified ReLU networks with $l_1$ and squared $l_2$ penalties attain, up to a logarithmic factor, the minimax rate of prediction of unknown $β$-smooth function.