Regularizing Neural Networks by Stochastically Training Layer Ensembles
This work addresses the need for stronger regularization in neural networks without increasing test-time computational cost, though it is incremental as it builds on existing stochastic methods like dropout.
The paper tackled the problem of improving neural network regularization by introducing STE layers, which train an ensemble of weight matrices with stochastic regularization and average outputs, resulting in consistent improvements on image classification tasks.
Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic regularization while explicitly averaging outputs. This provides stronger regularization with no additional computational cost at test time. We show consistent improvement on various image classification tasks using standard network topologies.