On the Implicit Bias of Dropout
This work addresses the theoretical understanding of dropout's effects for researchers in machine learning, but it is incremental as it focuses on a specific network type.
The paper tackled the problem of understanding the implicit bias of dropout in deep learning, showing that for single hidden-layer linear neural networks, dropout tends to equalize the norms of weight vectors and provides a complete characterization of the optimization landscape.
Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For single hidden-layer linear neural networks, we show that dropout tends to make the norm of incoming/outgoing weight vectors of all the hidden nodes equal. In addition, we provide a complete characterization of the optimization landscape induced by dropout.