What do CNNs Learn in the First Layer and Why? A Linear Systems Perspective
This work provides insights into the fundamental behavior of CNNs for researchers in deep learning, though it is incremental as it builds on prior observations of consistency.
The authors quantified the consistency of first-layer representations in CNNs by analyzing energy distributions, finding they are remarkably stable across various conditions and approach a whitening transformation. They derived an analytical formula for linear CNNs that fits nonlinear models like ResNet and VGG, showing the first layer performs approximate whitening.
It has previously been reported that the representation that is learned in the first layer of deep Convolutional Neural Networks (CNNs) is highly consistent across initializations and architectures. In this work, we quantify this consistency by considering the first layer as a filter bank and measuring its energy distribution. We find that the energy distribution is very different from that of the initial weights and is remarkably consistent across random initializations, datasets, architectures and even when the CNNs are trained with random labels. In order to explain this consistency, we derive an analytical formula for the energy profile of linear CNNs and show that this profile is mostly dictated by the second order statistics of image patches in the training set and it will approach a whitening transformation when the number of iterations goes to infinity. Finally, we show that this formula for linear CNNs also gives an excellent fit for the energy profiles learned by commonly used nonlinear CNNs such as ResNet and VGG, and that the first layer of these CNNs indeed perform approximate whitening of their inputs.