Dropout Inference with Non-Uniform Weight Scaling
This is an incremental improvement for neural network regularization, addressing specific scenarios in dropout-based inference.
The paper tackles the problem of dropout inference when some submodels behave like high-bias models, showing that non-uniform weight scaling provides a better approximation than standard methods.
Dropout as regularization has been used extensively to prevent overfitting for training neural networks. During training, units and their connections are randomly dropped, which could be considered as sampling many different submodels from the original model. At test time, weight scaling and Monte Carlo approximation are two widely applied approaches to approximate the outputs. Both approaches work well practically when all submodels are low-bias complex learners. However, in this work, we demonstrate scenarios where some submodels behave closer to high-bias models and a non-uniform weight scaling is a better approximation for inference.