Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networks
This work addresses the challenge of reliable OoD detection for deep learning practitioners, offering a simple modification that enhances performance, though it is incremental as it builds on existing methods like DDU.
The paper tackles the problem of improving out-of-distribution (OoD) detection in deep neural networks by proposing L2 normalization over feature space, which induces early Neural Collapse and achieves comparable or superior OoD detection scores and classification accuracy in a small fraction of the training time on the Deep Deterministic Uncertainty benchmark.
We propose a simple modification to standard ResNet architectures--L2 normalization over feature space--that substantially improves out-of-distribution (OoD) performance on the previously proposed Deep Deterministic Uncertainty (DDU) benchmark. We show that this change also induces early Neural Collapse (NC), an effect linked to better OoD performance. Our method achieves comparable or superior OoD detection scores and classification accuracy in a small fraction of the training time of the benchmark. Additionally, it substantially improves worst case OoD performance over multiple, randomly initialized models. Though we do not suggest that NC is the sole mechanism or a comprehensive explanation for OoD behaviour in deep neural networks (DNN), we believe NC's simple mathematical and geometric structure can provide a framework for analysis of this complex phenomenon in future work.