Log-Linear RNNs: Towards Recurrent Neural Networks with Flexible Prior Knowledge
This work addresses data sparsity and prior knowledge integration in RNNs for language modeling, representing an incremental improvement.
The authors tackled the problem of training data sparsity and inflexible prior knowledge in recurrent neural networks by introducing Log-Linear RNNs (LL-RNNs), which replace the softmax output layer with a log-linear layer, resulting in an important decrease in perplexity in French language modeling experiments.
We introduce LL-RNNs (Log-Linear RNNs), an extension of Recurrent Neural Networks that replaces the softmax output layer by a log-linear output layer, of which the softmax is a special case. This conceptually simple move has two main advantages. First, it allows the learner to combat training data sparsity by allowing it to model words (or more generally, output symbols) as complex combinations of attributes without requiring that each combination is directly observed in the training data (as the softmax does). Second, it permits the inclusion of flexible prior knowledge in the form of a priori specified modular features, where the neural network component learns to dynamically control the weights of a log-linear distribution exploiting these features. We conduct experiments in the domain of language modelling of French, that exploit morphological prior knowledge and show an important decrease in perplexity relative to a baseline RNN. We provide other motivating iillustrations, and finally argue that the log-linear and the neural-network components contribute complementary strengths to the LL-RNN: the LL aspect allows the model to incorporate rich prior knowledge, while the NN aspect, according to the "representation learning" paradigm, allows the model to discover novel combination of characteristics.