Phase Collapse in Neural Networks
This work addresses a fundamental challenge in neural network theory for researchers, providing insights into mechanisms beyond sparsity for improving class separation.
The paper tackles the problem of understanding how non-linearities and convolutional filters transform spatial variability into channel variability in deep convolutional classifiers, demonstrating that phase collapse of complex wavelet coefficients achieves classification accuracy comparable to ResNets, while thresholding operators degrade performance.
Deep convolutional classifiers linearly separate image classes and improve accuracy as depth increases. They progressively reduce the spatial dimension whereas the number of channels grows with depth. Spatial variability is therefore transformed into variability along channels. A fundamental challenge is to understand the role of non-linearities together with convolutional filters in this transformation. ReLUs with biases are often interpreted as thresholding operators that improve discrimination through sparsity. This paper demonstrates that it is a different mechanism called phase collapse which eliminates spatial variability while linearly separating classes. We show that collapsing the phases of complex wavelet coefficients is sufficient to reach the classification accuracy of ResNets of similar depths. However, replacing the phase collapses with thresholding operators that enforce sparsity considerably degrades the performance. We explain these numerical results by showing that the iteration of phase collapses progressively improves separation of classes, as opposed to thresholding non-linearities.