Deep Neural Network Models Trained With A Fixed Random Classifier Transfer Better Across Domains
This work addresses the challenge of domain transfer in deep learning, particularly for fine-grained image classification, by leveraging insights from neural collapse to enhance model generalization, though it is incremental as it builds on known phenomena.
The paper tackles the problem of improving transfer learning performance of deep neural networks across domains by fixing the last-layer classifier to an Equiangular Tight Frame (ETF) geometry, which enforces class separation and eliminates class covariance information. The result shows significant improvements, outperforming baseline methods by up to 22% and explicit covariance whitening methods by up to 19% on fine-grained image classification datasets.
The recently discovered Neural collapse (NC) phenomenon states that the last-layer weights of Deep Neural Networks (DNN), converge to the so-called Equiangular Tight Frame (ETF) simplex, at the terminal phase of their training. This ETF geometry is equivalent to vanishing within-class variability of the last layer activations. Inspired by NC properties, we explore in this paper the transferability of DNN models trained with their last layer weight fixed according to ETF. This enforces class separation by eliminating class covariance information, effectively providing implicit regularization. We show that DNN models trained with such a fixed classifier significantly improve transfer performance, particularly on out-of-domain datasets. On a broad range of fine-grained image classification datasets, our approach outperforms i) baseline methods that do not perform any covariance regularization (up to 22%), as well as ii) methods that explicitly whiten covariance of activations throughout training (up to 19%). Our findings suggest that DNNs trained with fixed ETF classifiers offer a powerful mechanism for improving transfer learning across domains.