Understanding How Nonlinear Layers Create Linearly Separable Features for Low-Dimensional Data
This work provides theoretical justification for empirical observations of linear separability in deep networks, bridging a gap in model interpretability and generalization for machine learning researchers.
The paper tackles the problem of understanding how deep networks learn linearly separable features by showing that a single nonlinear layer can transform low-dimensional data into linearly separable sets with high probability using random weights and quadratic activations, achieving this with network width scaling polynomially with intrinsic dimension rather than ambient dimension.
Deep neural networks have attained remarkable success across diverse classification tasks. Recent empirical studies have shown that deep networks learn features that are linearly separable across classes. However, these findings often lack rigorous justifications, even under relatively simple settings. In this work, we address this gap by examining the linear separation capabilities of shallow nonlinear networks. Specifically, inspired by the low intrinsic dimensionality of image data, we model inputs as a union of low-dimensional subspaces (UoS) and demonstrate that a single nonlinear layer can transform such data into linearly separable sets. Theoretically, we show that this transformation occurs with high probability when using random weights and quadratic activations. Notably, we prove this can be achieved when the network width scales polynomially with the intrinsic dimension of the data rather than the ambient dimension. Experimental results corroborate these theoretical findings and demonstrate that similar linear separation properties hold in practical scenarios beyond our analytical scope. This work bridges the gap between empirical observations and theoretical understanding of the separation capacity of nonlinear networks, offering deeper insights into model interpretability and generalization.