Information Geometry of Dropout Training
This provides theoretical insights for researchers in machine learning, though it is incremental as it builds on existing dropout analyses.
The paper tackles the problem of understanding dropout regularization in neural networks by analyzing its properties from an information geometry perspective, showing that dropout flattens the model manifold and corresponds to Fisher information regularization, with numerical experiments supporting these findings.
Dropout is one of the most popular regularization techniques in neural network training. Because of its power and simplicity of idea, dropout has been analyzed extensively and many variants have been proposed. In this paper, several properties of dropout are discussed in a unified manner from the viewpoint of information geometry. We showed that dropout flattens the model manifold and that their regularization performance depends on the amount of the curvature. Then, we showed that dropout essentially corresponds to a regularization that depends on the Fisher information, and support this result from numerical experiments. Such a theoretical analysis of the technique from a different perspective is expected to greatly assist in the understanding of neural networks, which are still in their infancy.