The role of class encoding in neural collapse
Provides theoretical insights into the role of label encoding in neural collapse, relevant for understanding and improving deep learning classification models.
This paper investigates how label encoding affects neural collapse in neural networks, showing that for one-hot encoded labels with balanced data, the class features transition from a simplex equiangular tight frame to an orthogonal frame as bias regularization increases, and that bias centers the labels for arbitrary encodings.
Neural collapse is a structural property of the last-hidden-layer activations in neural network classification models, when trained beyond a zero classification error. In this work, we explore the role of label encoding in neural collapse by relying on the unrestricted feature model with mean squared error training loss. We demonstrate that, for one-hot encoded labels and balanced data, the uncentered mean features associated with each class transition from a simplex equiangular tight frame to an orthogonal frame when increasing the bias regularization coefficient associated with the final classifier. These structures are reminiscent of the orthogonal frame structure of one-hot encoded labels. For any arbitrary encoding, we also show that the final classifier's bias aims at centering the labels, compensating for the discrepancy between the global mean of the labels and the origin. We further discuss the role of the encoding in other neural collapse properties.