Stochastic Modified Equations and Dynamics of Dropout Algorithm
This work provides theoretical insights into dropout's tendency to find flatter minima, which is incremental but addresses a key gap in regularization techniques for neural networks.
The paper tackled the problem of understanding dropout's mechanism for generalization by deriving stochastic modified equations to approximate its dynamics, and empirically demonstrated universal relations between variance and flatness of minima throughout training.
Dropout is a widely utilized regularization technique in the training of neural networks, nevertheless, its underlying mechanism and its impact on achieving good generalization abilities remain poorly understood. In this work, we derive the stochastic modified equations for analyzing the dynamics of dropout, where its discrete iteration process is approximated by a class of stochastic differential equations. In order to investigate the underlying mechanism by which dropout facilitates the identification of flatter minima, we study the noise structure of the derived stochastic modified equation for dropout. By drawing upon the structural resemblance between the Hessian and covariance through several intuitive approximations, we empirically demonstrate the universal presence of the inverse variance-flatness relation and the Hessian-variance relation, throughout the training process of dropout. These theoretical and empirical findings make a substantial contribution to our understanding of the inherent tendency of dropout to locate flatter minima.