FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding
This work addresses the need for faster and more efficient sequence encoding in NLP, offering a modular solution that enhances Transformer models, though it is incremental in improving existing methods.
The paper tackled the problem of slow sequence encoding in latent tree induction by proposing FastTrees, a parallelizable neural module that achieved competitive or superior performance to ON-LSTM on tasks like language modeling and outperformed state-of-the-art models by +4% on logical inference and +8% on mathematical language understanding.
Inducing latent tree structures from sequential data is an emerging trend in the NLP research landscape today, largely popularized by recent methods such as Gumbel LSTM and Ordered Neurons (ON-LSTM). This paper proposes FASTTREES, a new general purpose neural module for fast sequence encoding. Unlike most previous works that consider recurrence to be necessary for tree induction, our work explores the notion of parallel tree induction, i.e., imbuing our model with hierarchical inductive biases in a parallelizable, non-autoregressive fashion. To this end, our proposed FASTTREES achieves competitive or superior performance to ON-LSTM on four well-established sequence modeling tasks, i.e., language modeling, logical inference, sentiment analysis and natural language inference. Moreover, we show that the FASTTREES module can be applied to enhance Transformer models, achieving performance gains on three sequence transduction tasks (machine translation, subject-verb agreement and mathematical language understanding), paving the way for modular tree induction modules. Overall, we outperform existing state-of-the-art models on logical inference tasks by +4% and mathematical language understanding by +8%.