A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalization
This work provides theoretical insights into neural network optimization and generalization for classification tasks, though it appears incremental to the original Neural Collapse framework.
The authors extended the Neural Collapse Phenomenon by proving the Generalized Neural Collapse hypothesis, revealing a Grassmannian Frame structure that maximizes feature separation on a sphere without requiring high dimensionality. They discovered the Symmetric Generalization phenomenon through experiments showing models with different Grassmannian Frames have varying performance, though the underlying reasons remain unexplained.
In this paper, we extend original Neural Collapse Phenomenon by proving Generalized Neural Collapse hypothesis. We obtain Grassmannian Frame structure from the optimization and generalization of classification. This structure maximally separates features of every two classes on a sphere and does not require a larger feature dimension than the number of classes. Out of curiosity about the symmetry of Grassmannian Frame, we conduct experiments to explore if models with different Grassmannian Frames have different performance. As a result, we discover the Symmetric Generalization phenomenon. We provide a theorem to explain Symmetric Generalization of permutation. However, the question of why different directions of features can lead to such different generalization is still open for future investigation.