Preventing RNN from Using Sequence Length as a Feature
This addresses a critical issue for practitioners using RNNs in text classification, as it prevents models from relying on trivial features, though the solution is incremental.
The paper tackles the problem of recurrent neural networks (RNNs) incorrectly using sequence length differences as a classification feature, which leads to brittle models and misleading performance; it proposes a solution using weight decay regularization, demonstrating effectiveness on synthetic and real-world data.
Recurrent neural networks are deep learning topologies that can be trained to classify long documents. However, in our recent work, we found a critical problem with these cells: they can use the length differences between texts of different classes as a prominent classification feature. This has the effect of producing models that are brittle and fragile to concept drift, can provide misleading performances and are trivially explainable regardless of text content. This paper illustrates the problem using synthetic and real-world data and provides a simple solution using weight decay regularization.