Dropout as data augmentation
This provides a novel perspective on dropout for machine learning practitioners, though it is incremental in nature.
The paper tackles the problem of interpreting dropout as a form of data augmentation in the input space, showing that projecting dropout noise back to the input can generate augmented training data, with results indicating similar performance to standard dropout and a new noise scheme that improves outcomes without extra computational cost.
Dropout is typically interpreted as bagging a large number of models sharing parameters. We show that using dropout in a network can also be interpreted as a kind of data augmentation in the input space without domain knowledge. We present an approach to projecting the dropout noise within a network back into the input space, thereby generating augmented versions of the training data, and we show that training a deterministic network on the augmented samples yields similar results. Finally, we propose a new dropout noise scheme based on our observations and show that it improves dropout results without adding significant computational cost.