On learning an interpreted language with recurrent models
This addresses the problem of language understanding in AI, but it is incremental as it builds on existing models with specific constraints.
The study investigated whether recurrent neural networks (LSTM and GRU) can learn to understand language by generalizing to compositional interpretation, finding they succeed only under favorable conditions like a well-paced curriculum, extensive data, and left-to-right composition.
Can recurrent neural nets, inspired by human sequential data processing, learn to understand language? We construct simplified datasets reflecting core properties of natural language as modeled in formal syntax and semantics: recursive syntactic structure and compositionality. We find LSTM and GRU networks to generalise to compositional interpretation well, but only in the most favorable learning settings, with a well-paced curriculum, extensive training data, and left-to-right (but not right-to-left) composition.