Sample Variance Decay in Randomly Initialized ReLU Networks
This addresses a subtle initialization issue in deep neural networks, potentially improving training efficiency for practitioners, though it is an incremental refinement of existing initialization methods.
The paper investigates the decay of sample variance in randomly initialized deep ReLU networks, showing that even when total variance is preserved, sample variance decreases in later layers, which can affect training dynamics. It demonstrates that Batch Normalization eliminates this decay and suggests that preserving sample variance at initialization may improve training.
Before training a neural net, a classic rule of thumb is to randomly initialize the weights so the variance of activations is preserved across layers. This is traditionally interpreted using the total variance due to randomness in both weights \emph{and} samples. Alternatively, one can interpret the rule of thumb as preservation of the variance over samples for a fixed network. The two interpretations differ little for a shallow net, but the difference is shown to grow with depth for a deep ReLU net by decomposing the total variance into the network-averaged sum of the sample variance and square of the sample mean. We demonstrate that even when the total variance is preserved, the sample variance decays in the later layers through an analytical calculation in the limit of infinite network width, and numerical simulations for finite width. We show that Batch Normalization eliminates this decay and provide empirical evidence that preserving the sample variance instead of only the total variance at initialization time can have an impact on the training dynamics of a deep network.