LSTMs Compose (and Learn) Bottom-Up
This work provides insights into the learning process of LSTMs for hierarchical composition, which is incremental as it builds on existing knowledge about LSTM behavior in NLP.
The paper investigates how LSTM language models learn hierarchical structure, showing that their compositional behavior arises from bottom-up learning where constituent representations rely on effective representations of shorter children, with Decompositional Interdependence (DI) correlating with syntactic distance in English data.
Recent work in NLP shows that LSTM language models capture hierarchical structure in language data. In contrast to existing work, we consider the \textit{learning} process that leads to their compositional behavior. For a closer look at how an LSTM's sequential representations are composed hierarchically, we present a related measure of Decompositional Interdependence (DI) between word meanings in an LSTM, based on their gate interactions. We connect this measure to syntax with experiments on English language data, where DI is higher on pairs of words with lower syntactic distance. To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data. These synthetic experiments support a specific hypothesis about how hierarchical structures are discovered over the course of training: that LSTM constituent representations are learned bottom-up, relying on effective representations of their shorter children, rather than learning the longer-range relations independently from children.