Spectral Analysis of Latent Representations
This provides a tool for researchers and practitioners to analyze neural network layers during training, but it is incremental as it builds on existing spectral analysis methods.
The authors tackled the problem of analyzing neural network representations by proposing Layer Saturation, a metric based on spectral analysis that measures the proportion of eigenvalues needed to explain 99% of variance in latent representations, and showed it relates to generalization and predictive performance.
We propose a metric, Layer Saturation, defined as the proportion of the number of eigenvalues needed to explain 99% of the variance of the latent representations, for analyzing the learned representations of neural network layers. Saturation is based on spectral analysis and can be computed efficiently, making live analysis of the representations practical during training. We provide an outlook for future applications of this metric by outlining the behaviour of layer saturation in different neural architectures and problems. We further show that saturation is related to the generalization and predictive performance of neural networks.