Characterizing Well-Behaved vs. Pathological Deep Neural Networks
This work addresses the challenge of understanding and mitigating training issues in deep learning for researchers and practitioners, offering insights into network design, but it is incremental as it builds on existing theories of initialization and architecture.
The paper tackles the problem of characterizing deep neural networks at initialization to distinguish well-behaved from pathological cases, finding that feedforward networks inevitably develop pathologies due to multiplicative layer composition, while residual networks with batch normalization avoid pathologies through power-law behavior induced by skip-connections.
We introduce a novel approach, requiring only mild assumptions, for the characterization of deep neural networks at initialization. Our approach applies both to fully-connected and convolutional networks and easily incorporates batch normalization and skip-connections. Our key insight is to consider the evolution with depth of statistical moments of signal and noise, thereby characterizing the presence or absence of pathologies in the hypothesis space encoded by the choice of hyperparameters. We establish: (i) for feedforward networks, with and without batch normalization, the multiplicativity of layer composition inevitably leads to ill-behaved moments and pathologies; (ii) for residual networks with batch normalization, on the other hand, skip-connections induce power-law rather than exponential behaviour, leading to well-behaved moments and no pathology.