Modeling Hierarchical Structures with Continuous Recursive Neural Networks
This addresses a bottleneck in natural language processing for models requiring hierarchical structure induction, offering a more stable training method compared to prior extensions.
The paper tackles the problem of Recursive Neural Networks (RvNNs) being unable to induce latent hierarchical structure from plain text, proposing Continuous Recursive Neural Network (CRvNN) as a backpropagation-friendly alternative. It demonstrates strong performance on synthetic tasks like logical inference and ListOps, and comparable or better results on real-world tasks such as sentiment analysis and natural language inference.
Recursive Neural Networks (RvNNs), which compose sequences according to their underlying hierarchical syntactic structure, have performed well in several natural language processing tasks compared to similar models without structural biases. However, traditional RvNNs are incapable of inducing the latent structure in a plain text sequence on their own. Several extensions have been proposed to overcome this limitation. Nevertheless, these extensions tend to rely on surrogate gradients or reinforcement learning at the cost of higher bias or variance. In this work, we propose Continuous Recursive Neural Network (CRvNN) as a backpropagation-friendly alternative to address the aforementioned limitations. This is done by incorporating a continuous relaxation to the induced structure. We demonstrate that CRvNN achieves strong performance in challenging synthetic tasks such as logical inference and ListOps. We also show that CRvNN performs comparably or better than prior latent structure models on real-world tasks such as sentiment analysis and natural language inference.