The Double Descent Behavior in Two Layer Neural Network for Binary Classification
This work provides theoretical insights into a counterintuitive behavior in neural networks, which is incremental as it builds on prior observations of double descent.
The authors tackled the double descent phenomenon in test error for two-layer ReLU neural networks in binary classification, using the Convex Gaussian Min Max Theorem to analyze how error varies with model size, quantified by the sample-to-dimension ratio.
Recent studies observed a surprising concept on model test error called the double descent phenomenon, where the increasing model complexity decreases the test error first and then the error increases and decreases again. To observe this, we work on a two layer neural network model with a ReLU activation function designed for binary classification under supervised learning. Our aim is to observe and investigate the mathematical theory behind the double descent behavior of model test error for varying model sizes. We quantify the model size by the ratio of number of training samples to the dimension of the model. Due to the complexity of the empirical risk minimization procedure, we use the Convex Gaussian Min Max Theorem to find a suitable candidate for the global training loss.