Understanding Feature Selection and Feature Memorization in Recurrent Neural Networks
This work addresses a fundamental challenge in sequence learning for researchers and practitioners in machine learning, though it is incremental in building on existing RNN models.
The paper introduces the Flagged-1-Bit test to analyze recurrent neural networks' sequence learning capabilities, revealing a conflict between feature selection and memorization that is resolved by gating mechanisms like LSTM or increasing state dimensions in Vanilla RNN.
In this paper, we propose a test, called Flagged-1-Bit (F1B) test, to study the intrinsic capability of recurrent neural networks in sequence learning. Four different recurrent network models are studied both analytically and experimentally using this test. Our results suggest that in general there exists a conflict between feature selection and feature memorization in sequence learning. Such a conflict can be resolved either using a gating mechanism as in LSTM, or by increasing the state dimension as in Vanilla RNN. Gated models resolve this conflict by adaptively adjusting their state-update equations, whereas Vanilla RNN resolves this conflict by assigning different dimensions different tasks. Insights into feature selection and memorization in recurrent networks are given.