On the interplay between data structure and loss function in classification problems
This work addresses a foundational puzzle in machine learning about generalization in overparametrized models, providing theoretical insights that are incremental but specific to structured data scenarios.
The authors tackled the problem of understanding how data structure and loss functions affect generalization in overparametrized models, showing that logistic loss outperforms mean-squared loss in the overparametrized regime, with performance gaps widening for easier tasks, as validated on MNIST and CIFAR10.
One of the central puzzles in modern machine learning is the ability of heavily overparametrized models to generalize well. Although the low-dimensional structure of typical datasets is key to this behavior, most theoretical studies of overparametrization focus on isotropic inputs. In this work, we instead consider an analytically tractable model of structured data, where the input covariance is built from independent blocks allowing us to tune the saliency of low-dimensional structures and their alignment with respect to the target function. Using methods from statistical physics, we derive a precise asymptotic expression for the train and test error achieved by random feature models trained to classify such data, which is valid for any convex loss function. We study in detail how the data structure affects the double descent curve, and show that in the over-parametrized regime, its impact is greater for logistic loss than for mean-squared loss: the easier the task, the wider the gap in performance at the advantage of the logistic loss. Our insights are confirmed by numerical experiments on MNIST and CIFAR10.