Fast convergence rates of deep neural networks for classification
This work provides theoretical guarantees for DNN convergence in classification, which is important for researchers and practitioners in machine learning, though it is incremental as it builds on existing theory.
The paper tackles the problem of deriving fast convergence rates for deep neural network classifiers using hinge loss and cross-entropy, showing that with carefully selected architectures, they achieve fast rates under various true model conditions, such as smooth decision boundaries and margin conditions, with numerical studies confirming the theoretical results.
We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., the probability of inputs near the decision boundary is small). We show that the DNN classifier learned using the hinge loss achieves fast rate convergences for all three cases provided that the architecture (i.e., the number of layers, number of nodes and sparsity). is carefully selected. An important implication is that DNN architectures are very flexible for use in various cases without much modification. In addition, we consider a DNN classifier learned by minimizing the cross-entropy, and show that the DNN classifier achieves a fast convergence rate under the condition that the conditional class probabilities of most data are sufficiently close to either 1 or zero. This assumption is not unusual for image recognition because human beings are extremely good at recognizing most images. To confirm our theoretical explanation, we present the results of a small numerical study conducted to compare the hinge loss and cross-entropy.