Interpolating Discriminant Functions in High-Dimensional Gaussian Latent Mixtures
This work addresses classification accuracy in high-dimensional noisy data, offering a corrected method for practitioners, though it is incremental as it builds on existing linear regression results.
The paper tackles binary classification in high-dimensional Gaussian latent mixtures by using a generalized least squares estimator to estimate the optimal separating hyperplane, which interpolates training data; it shows that a naive intercept estimate fails, but a corrected method with a hold-out sample achieves minimax optimality.
This paper considers binary classification of high-dimensional features under a postulated model with a low-dimensional latent Gaussian mixture structure and non-vanishing noise. A generalized least squares estimator is used to estimate the direction of the optimal separating hyperplane. The estimated hyperplane is shown to interpolate on the training data. While the direction vector can be consistently estimated as could be expected from recent results in linear regression, a naive plug-in estimate fails to consistently estimate the intercept. A simple correction, that requires an independent hold-out sample, renders the procedure minimax optimal in many scenarios. The interpolation property of the latter procedure can be retained, but surprisingly depends on the way the labels are encoded.