Convergence of Deep ReLU Networks
This provides mathematical insights for designing deep residual networks, addressing a foundational issue in neural network theory, but it is incremental as it builds on existing convergence studies.
The paper tackles the problem of understanding convergence in deep ReLU networks as depth increases to infinity, establishing necessary conditions (weight matrices converging to identity and bias vectors to zero) and sufficient conditions for pointwise convergence.
We explore convergence of deep neural networks with the popular ReLU activation function, as the depth of the networks tends to infinity. To this end, we introduce the notion of activation domains and activation matrices of a ReLU network. By replacing applications of the ReLU activation function by multiplications with activation matrices on activation domains, we obtain an explicit expression of the ReLU network. We then identify the convergence of the ReLU networks as convergence of a class of infinite products of matrices. Sufficient and necessary conditions for convergence of these infinite products of matrices are studied. As a result, we establish necessary conditions for ReLU networks to converge that the sequence of weight matrices converges to the identity matrix and the sequence of the bias vectors converges to zero as the depth of ReLU networks increases to infinity. Moreover, we obtain sufficient conditions in terms of the weight matrices and bias vectors at hidden layers for pointwise convergence of deep ReLU networks. These results provide mathematical insights to the design strategy of the well-known deep residual networks in image classification.