Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?
This provides theoretical insight into why depth matters in neural networks and their superiority over kernel methods, addressing a foundational problem in nonparametric regression for machine learning researchers.
The paper tackles the problem of neural networks adaptively estimating functions with heterogeneous smoothness, showing that with only regularization tuning, a parallel NN variant achieves estimation error arbitrarily close to minimax rates for Besov and BV classes, with exponential improvement as depth increases.
We study the theory of neural network (NN) from the lens of classical nonparametric regression problems with a focus on NN's ability to adaptively estimate functions with heterogeneous smoothness -- a property of functions in Besov or Bounded Variation (BV) classes. Existing work on this problem requires tuning the NN architecture based on the function spaces and sample size. We consider a "Parallel NN" variant of deep ReLU networks and show that the standard $\ell_2$ regularization is equivalent to promoting the $\ell_p$-sparsity ($0<p<1$) in the coefficient vector of an end-to-end learned function bases, i.e., a dictionary. Using this equivalence, we further establish that by tuning only the regularization factor, such parallel NN achieves an estimation error arbitrarily close to the minimax rates for both the Besov and BV classes. Notably, it gets exponentially closer to minimax optimal as the NN gets deeper. Our research sheds new lights on why depth matters and how NNs are more powerful than kernel methods.