Neural (Tangent Kernel) Collapse
This work provides a theoretical foundation for understanding symmetry in deep learning features, which is incremental as it builds on existing NTK and NC concepts.
The paper tackles the problem of linking Neural Tangent Kernel (NTK) dynamics with Neural Collapse (NC) in deep neural networks, showing that under a block-structured NTK assumption, NC emerges during training with mean squared loss, supported by experiments on three architectures and datasets.
This work bridges two important concepts: the Neural Tangent Kernel (NTK), which captures the evolution of deep neural networks (DNNs) during training, and the Neural Collapse (NC) phenomenon, which refers to the emergence of symmetry and structure in the last-layer features of well-trained classification DNNs. We adopt the natural assumption that the empirical NTK develops a block structure aligned with the class labels, i.e., samples within the same class have stronger correlations than samples from different classes. Under this assumption, we derive the dynamics of DNNs trained with mean squared (MSE) loss and break them into interpretable phases. Moreover, we identify an invariant that captures the essence of the dynamics, and use it to prove the emergence of NC in DNNs with block-structured NTK. We provide large-scale numerical experiments on three common DNN architectures and three benchmark datasets to support our theory.