Occam's Gates
This addresses regularization and interpretability issues in RNNs for sequence classification, but it is incremental as it builds on existing gating mechanisms.
The paper tackles overfitting in attention-based RNN models by introducing an L1 penalty on gating unit activations, resulting in reduced overfitting and improved interpretability across tasks like sentiment analysis and question answering.
We present a complimentary objective for training recurrent neural networks (RNN) with gating units that helps with regularization and interpretability of the trained model. Attention-based RNN models have shown success in many difficult sequence to sequence classification problems with long and short term dependencies, however these models are prone to overfitting. In this paper, we describe how to regularize these models through an L1 penalty on the activation of the gating units, and show that this technique reduces overfitting on a variety of tasks while also providing to us a human-interpretable visualization of the inputs used by the network. These tasks include sentiment analysis, paraphrase recognition, and question answering.