Capacity allocation through neural network layers
This work provides theoretical insights into how neural networks distribute modeling capacity, which is incremental for researchers in deep learning theory.
The paper extends capacity allocation analysis to neural networks with non-linear layers, showing that under certain hypotheses, it reduces to linear allocation in an extended input space, and introduces layer decoupling to explain capacity propagation, recovering known results like effective receptive field sizes and why ResNets avoid shattering.
Capacity analysis has been recently introduced as a way to analyze how linear models distribute their modelling capacity across the input space. In this paper, we extend the notion of capacity allocation to the case of neural networks with non-linear layers. We show that under some hypotheses the problem is equivalent to linear capacity allocation, within some extended input space that factors in the non-linearities. We introduce the notion of layer decoupling, which quantifies the degree to which a non-linear activation decouples its outputs, and show that it plays a central role in capacity allocation through layers. In the highly non-linear limit where decoupling is total, we show that the propagation of capacity throughout the layers follows a simple markovian rule, which turns into a diffusion PDE in the limit of deep networks with residual layers. This allows us to recover some known results about deep neural networks, such as the size of the effective receptive field, or why ResNets avoid the shattering problem.