Learning curves of generic features maps for realistic datasets with a teacher-student model
This work addresses the limitation of teacher-student models in handling realistic data for researchers in machine learning theory, though it is incremental as it builds on existing frameworks.
The authors tackled the problem of extending teacher-student models to capture learning curves for realistic datasets by introducing a Gaussian covariate generalization with generic feature maps, proving rigorous formulas for asymptotic training loss and generalization error, and demonstrating its applicability to kernel regression, classification, and neural network features.
Teacher-student models provide a framework in which the typical-case performance of high-dimensional supervised learning can be described in closed form. The assumptions of Gaussian i.i.d. input data underlying the canonical teacher-student model may, however, be perceived as too restrictive to capture the behaviour of realistic data sets. In this paper, we introduce a Gaussian covariate generalisation of the model where the teacher and student can act on different spaces, generated with fixed, but generic feature maps. While still solvable in a closed form, this generalization is able to capture the learning curves for a broad range of realistic data sets, thus redeeming the potential of the teacher-student framework. Our contribution is then two-fold: First, we prove a rigorous formula for the asymptotic training loss and generalisation error. Second, we present a number of situations where the learning curve of the model captures the one of a realistic data set learned with kernel regression and classification, with out-of-the-box feature maps such as random projections or scattering transforms, or with pre-learned ones - such as the features learned by training multi-layer neural networks. We discuss both the power and the limitations of the framework.