Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
This addresses the challenge of improving generalization in RNNs for tasks like language modeling and sequential data processing, though it appears incremental as it builds on dropout and stochastic depth concepts.
The authors tackled the problem of regularizing recurrent neural networks (RNNs) by proposing zoneout, which randomly preserves hidden activations instead of dropping them, leading to significant performance improvements across tasks including competitive results on language modeling datasets and state-of-the-art results on permuted sequential MNIST when combined with recurrent batch normalization.
We propose zoneout, a novel method for regularizing RNNs. At each timestep, zoneout stochastically forces some hidden units to maintain their previous values. Like dropout, zoneout uses random noise to train a pseudo-ensemble, improving generalization. But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks. We perform an empirical investigation of various RNN regularizers, and find that zoneout gives significant performance improvements across tasks. We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization yields state-of-the-art results on permuted sequential MNIST.