The Space Between: On Folding, Symmetries and Sampling
This work addresses the understanding of neural network symmetries and generalization for researchers in machine learning, though it appears incremental as it builds on prior folding measures.
The authors tackled the problem of quantifying space folding in neural networks by generalizing a measure to various activation functions and analyzing its properties, leading to an efficient sampling strategy and a novel regularization scheme that encourages higher folding values to improve generalization.
Recent findings suggest that consecutive layers of neural networks with the ReLU activation function \emph{fold} the input space during the learning process. While many works hint at this phenomenon, an approach to quantify the folding was only recently proposed by means of a space folding measure based on Hamming distance in the ReLU activation space. We generalize this measure to a wider class of activation functions through introduction of equivalence classes of input data, analyse its mathematical and computational properties and come up with an efficient sampling strategy for its implementation. Moreover, it has been observed that space folding values increase with network depth when the generalization error is low, but decrease when the error increases. This underpins that learned symmetries in the data manifold (e.g., invariance under reflection) become visible in terms of space folds, contributing to the network's generalization capacity. Inspired by these findings, we outline a novel regularization scheme that encourages the network to seek solutions characterized by higher folding values.