An ETF view of Dropout regularization
This provides theoretical insight into a widely used regularization technique in machine learning, though it is incremental as it builds on existing dropout methods.
The paper tackles the problem of understanding why dropout regularization works in deep learning by providing a new interpretation from frame theory, showing that dropout leads to an equiangular tight frame (ETF) structure in linear encoders of autoencoders.
Dropout is a popular regularization technique in deep learning. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain family of autoencoders with a linear encoder, optimizing the encoder with dropout regularization leads to an equiangular tight frame (ETF). Since this optimization is non-convex, we add another regularization that promotes such structures by minimizing the cross-correlation between filters in the network. We demonstrate its applicability in convolutional and fully connected layers in both feed-forward and recurrent networks. All these results suggest that there is indeed a relationship between dropout and ETF structure of the regularized linear operations.