Super-fast rates of convergence for Neural Networks Classifiers under the Hard Margin Condition
This provides theoretical guarantees for neural network classifiers in scenarios with well-separated data, which is incremental as it extends existing convergence rate analyses to more specific conditions.
The paper tackles the binary classification problem using deep neural networks under the hard-margin condition, showing that they can achieve excess risk bounds of order O(n^{-α}) for arbitrarily large α, indicating extremely fast convergence rates.
We study the classical binary classification problem for hypothesis spaces of Deep Neural Networks (DNNs) with ReLU activation under Tsybakov's low-noise condition with exponent $q>0$, and its limit-case $q\to\infty$ which we refer to as the "hard-margin condition". We show that DNNs which minimize the empirical risk with square loss surrogate and $\ell_p$ penalty can achieve finite-sample excess risk bounds of order $\mathcal{O}\left(n^{-α}\right)$ for arbitrarily large $α>0$ under the hard-margin condition, provided that the regression function $η$ is sufficiently smooth. The proof relies on a novel decomposition of the excess risk which might be of independent interest.