Sup-Norm Convergence of Deep Neural Network Estimator for Nonparametric Regression by Adversarial Training
This addresses a theoretical bottleneck in deep learning for nonparametric regression, offering a method to improve convergence guarantees, though it appears incremental as it builds on existing adversarial training concepts.
The paper tackles the problem of achieving sup-norm convergence for deep neural network estimators in nonparametric regression, which is difficult with least-squares training, and shows that a novel adversarial training scheme with correction enables these estimators to achieve the optimal rate in the sup-norm sense, supported by experiments.
We show the sup-norm convergence of deep neural network estimators with a novel adversarial training scheme. For the nonparametric regression problem, it has been shown that an estimator using deep neural networks can achieve better performances in the sense of the $L2$-norm. In contrast, it is difficult for the neural estimator with least-squares to achieve the sup-norm convergence, due to the deep structure of neural network models. In this study, we develop an adversarial training scheme and investigate the sup-norm convergence of deep neural network estimators. First, we find that ordinary adversarial training makes neural estimators inconsistent. Second, we show that a deep neural network estimator achieves the optimal rate in the sup-norm sense by the proposed adversarial training with correction. We extend our adversarial training to general setups of a loss function and a data-generating function. Our experiments support the theoretical findings.