Dropout: Explicit Forms and Capacity Control
This work provides theoretical insights into dropout's regularization effects, benefiting researchers in machine learning by offering generalization bounds for matrix completion and deep learning tasks.
The paper investigates the capacity control provided by dropout in machine learning, showing that it induces data-dependent regularizers for matrix completion and deep neural networks, leading to concrete generalization error bounds validated on datasets like MovieLens, MNIST, and Fashion-MNIST.
We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a data-dependent regularizer that, in expectation, equals the weighted trace-norm of the product of the factors. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, MNIST, and Fashion-MNIST.