Beam Tree Recursive Cells
This work addresses structure-sensitive tasks in machine learning, such as logical inference, but is incremental as it builds on existing Recursive Neural Network frameworks.
The authors tackled the problem of extending Recursive Neural Networks with beam search for latent structure induction, proposing Beam Tree Recursive Cell (BT-Cell) and a relaxation for better gradient propagation, achieving near-perfect performance on synthetic tasks like ListOps and comparable results on realistic data.
We propose Beam Tree Recursive Cell (BT-Cell) - a backpropagation-friendly framework to extend Recursive Neural Networks (RvNNs) with beam search for latent structure induction. We further extend this framework by proposing a relaxation of the hard top-k operators in beam search for better propagation of gradient signals. We evaluate our proposed models in different out-of-distribution splits in both synthetic and realistic data. Our experiments show that BTCell achieves near-perfect performance on several challenging structure-sensitive synthetic tasks like ListOps and logical inference while maintaining comparable performance in realistic data against other RvNN-based models. Additionally, we identify a previously unknown failure case for neural models in generalization to unseen number of arguments in ListOps. The code is available at: https://github.com/JRC1995/BeamTreeRecursiveCells.