Feature Space Saturation during Training
This work provides a tool for analyzing neural network layers, which could help researchers optimize architectures, but it is incremental as it builds on existing variance-based methods.
The authors tackled the problem of understanding information processing in neural networks by introducing layer saturation, a metric derived from the eigenspace of a layer's variance matrix, and demonstrated that adjusting network parameters like input resolution can improve performance by distributing inference more evenly.
We propose layer saturation - a simple, online-computable method for analyzing the information processing in neural networks. First, we show that a layer's output can be restricted to the eigenspace of its variance matrix without performance loss. We propose a computationally lightweight method for approximating the variance matrix during training. From the dimension of its lossless eigenspace we derive layer saturation - the ratio between the eigenspace dimension and layer width. We show that saturation seems to indicate which layers contribute to network performance. We demonstrate how to alter layer saturation in a neural network by changing network depth, filter sizes and input resolution. Furthermore, we show that well-chosen input resolution increases network performance by distributing the inference process more evenly across the network.