Latent Point Collapse on a Low Dimensional Embedding in Deep Neural Network Classifiers
This addresses the challenge of enhancing generalization and robustness in deep learning classifiers, though it is incremental as it builds on existing methods for latent space optimization.
The paper tackles the problem of improving class separability and robustness in deep neural network classifiers by inducing latent point collapse, which compresses same-class representations into single points, resulting in substantial improvements in discriminative feature embeddings and remarkable gains in robustness to input perturbations.
The configuration of latent representations plays a critical role in determining the performance of deep neural network classifiers. In particular, the emergence of well-separated class embeddings in the latent space has been shown to improve both generalization and robustness. In this paper, we propose a method to induce the collapse of latent representations belonging to the same class into a single point, which enhances class separability in the latent space while enforcing Lipschitz continuity in the network. We demonstrate that this phenomenon, which we call \textit{latent point collapse}, is achieved by adding a strong $L_2$ penalty on the penultimate-layer representations and is the result of a push-pull tension developed with the cross-entropy loss function. In addition, we show the practical utility of applying this compressing loss term to the latent representations of a low-dimensional linear penultimate layer. The proposed approach is straightforward to implement and yields substantial improvements in discriminative feature embeddings, along with remarkable gains in robustness to input perturbations.