Surprisal-Driven Zoneout
This provides a novel regularization technique for recurrent neural networks that improves sequence modeling performance, though it appears incremental relative to existing zoneout methods.
The paper tackles the problem of regularizing recurrent neural networks by proposing surprisal-driven zoneout, which adaptively maintains previous neuron states when prediction error is small. This method achieved state-of-the-art performance with 1.31 bits per character on the Hutter Prize Wikipedia dataset, significantly closing the gap to highly-engineered compression methods.
We propose a novel method of regularization for recurrent neural networks called suprisal-driven zoneout. In this method, states zoneout (maintain their previous value rather than updating), when the suprisal (discrepancy between the last state's prediction and target) is small. Thus regularization is adaptive and input-driven on a per-neuron basis. We demonstrate the effectiveness of this idea by achieving state-of-the-art bits per character of 1.31 on the Hutter Prize Wikipedia dataset, significantly reducing the gap to the best known highly-engineered compression methods.