Learning curves for the multi-class teacher-student perceptron
This work addresses a gap in theoretical understanding for multi-class classification in high-dimensional settings, which is incremental as it extends existing binary-class results to multi-class scenarios.
The paper derived asymptotic expressions for the generalization errors of both Bayes-optimal estimation and empirical risk minimization in the multi-class teacher-student perceptron, filling a gap in high-dimensional learning theory. It found that regularized cross-entropy minimization yields close-to-optimal accuracy for Gaussian teacher weights and identified a first-order phase transition in Bayes-optimal performance for a binary teacher.
One of the most classical results in high-dimensional learning theory provides a closed-form expression for the generalisation error of binary classification with the single-layer teacher-student perceptron on i.i.d. Gaussian inputs. Both Bayes-optimal estimation and empirical risk minimisation (ERM) were extensively analysed for this setting. At the same time, a considerable part of modern machine learning practice concerns multi-class classification. Yet, an analogous analysis for the corresponding multi-class teacher-student perceptron was missing. In this manuscript we fill this gap by deriving and evaluating asymptotic expressions for both the Bayes-optimal and ERM generalisation errors in the high-dimensional regime. For Gaussian teacher weights, we investigate the performance of ERM with both cross-entropy and square losses, and explore the role of ridge regularisation in approaching Bayes-optimality. In particular, we observe that regularised cross-entropy minimisation yields close-to-optimal accuracy. Instead, for a binary teacher we show that a first-order phase transition arises in the Bayes-optimal performance.