Semi-Supervised Deep Sobolev Regression: Estimation and Variable Selection by ReQU Neural Network
This work addresses the challenge of simultaneous regression and gradient estimation in semi-supervised learning, with applications like variable selection, though it appears incremental as it builds on existing neural network and regularization methods.
The authors tackled the problem of nonparametric estimation of regression functions and their gradients by proposing SDORE, a semi-supervised deep Sobolev regressor using ReQU neural networks, which achieves minimax optimal convergence rates and demonstrates effectiveness in numerical simulations.
We propose SDORE, a Semi-supervised Deep Sobolev Regressor, for the nonparametric estimation of the underlying regression function and its gradient. SDORE employs deep ReQU neural networks to minimize the empirical risk with gradient norm regularization, allowing the approximation of the regularization term by unlabeled data. Our study includes a thorough analysis of the convergence rates of SDORE in $L^{2}$-norm, achieving the minimax optimality. Further, we establish a convergence rate for the associated plug-in gradient estimator, even in the presence of significant domain shift. These theoretical findings offer valuable insights for selecting regularization parameters and determining the size of the neural network, while showcasing the provable advantage of leveraging unlabeled data in semi-supervised learning. To the best of our knowledge, SDORE is the first provable neural network-based approach that simultaneously estimates the regression function and its gradient, with diverse applications such as nonparametric variable selection. The effectiveness of SDORE is validated through an extensive range of numerical simulations.