Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks
This work addresses the need for theoretical understanding of model simplification techniques in deep learning, though it appears incremental as it builds on existing mean field theory with large deviation analysis.
The paper tackled the problem of analyzing deviations from mean field solutions in finite-sized deep neural networks under parameter perturbations like weight sparsification and binarization, finding that random networks with ReLU activation are more robust than those with sign activation, as reflected in the simplicity of the functions they generate.
Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The parameter perturbations investigated include weight sparsification (dilution) and binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are more robust to parameter perturbations with respect to their counterparts with sign activation, which arguably is reflected in the simplicity of the functions they generate.