Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)
This work addresses the problem of understanding robustness in deep learning for researchers, providing theoretical insights that refine prior results but are incremental in nature.
The paper investigates how width, depth, and initialization affect the average robustness of deep neural networks, finding that width improves robustness in over-parameterized settings but harms it in under-parameterized ones, while depth's impact varies with initialization and training regime.
We study the average robustness notion in deep neural networks in (selected) wide and narrow, deep and shallow, as well as lazy and non-lazy training settings. We prove that in the under-parameterized setting, width has a negative effect while it improves robustness in the over-parameterized setting. The effect of depth closely depends on the initialization and the training mode. In particular, when initialized with LeCun initialization, depth helps robustness with the lazy training regime. In contrast, when initialized with Neural Tangent Kernel (NTK) and He-initialization, depth hurts the robustness. Moreover, under the non-lazy training regime, we demonstrate how the width of a two-layer ReLU network benefits robustness. Our theoretical developments improve the results by [Huang et al. NeurIPS21; Wu et al. NeurIPS21] and are consistent with [Bubeck and Sellke NeurIPS21; Bubeck et al. COLT21].