Robust deep learning from weakly dependent data
This work addresses robust deep learning for weakly dependent data, which is incremental as it extends theoretical guarantees to more realistic, unbounded scenarios beyond common assumptions.
The paper tackles robust deep learning with unbounded loss functions and inputs under weak data dependence, establishing non-asymptotic bounds for the expected excess risk of deep neural network estimators. It shows that with sufficiently smooth target functions, the convergence rate for exponentially strongly mixing data matches or approaches that of i.i.d. samples, and simulations demonstrate robust estimators outperform least squares in heavy-tailed error models.
Recent developments on deep learning established some theoretical properties of deep neural networks estimators. However, most of the existing works on this topic are restricted to bounded loss functions or (sub)-Gaussian or bounded input. This paper considers robust deep learning from weakly dependent observations, with unbounded loss function and unbounded input/output. It is only assumed that the output variable has a finite $r$ order moment, with $r >1$. Non asymptotic bounds for the expected excess risk of the deep neural network estimator are established under strong mixing, and $ψ$-weak dependence assumptions on the observations. We derive a relationship between these bounds and $r$, and when the data have moments of any order (that is $r=\infty$), the convergence rate is close to some well-known results. When the target predictor belongs to the class of Hölder smooth functions with sufficiently large smoothness index, the rate of the expected excess risk for exponentially strongly mixing data is close to or as same as those for obtained with i.i.d. samples. Application to robust nonparametric regression and robust nonparametric autoregression are considered. The simulation study for models with heavy-tailed errors shows that, robust estimators with absolute loss and Huber loss function outperform the least squares method.