Differentiable Reasoning over Long Stories -- Assessing Systematic Generalisation in Neural Models
This addresses the issue of neural networks failing on out-of-distribution data for natural language understanding, but it is incremental as it compares existing model types on a known benchmark.
The paper tackled the problem of systematic generalization in neural models on long stories, finding that a modified recurrent neural network achieved surprisingly accurate results across tasks, outperforming a modified graph neural network, which produced more robust models.
Contemporary neural networks have achieved a series of developments and successes in many aspects; however, when exposed to data outside the training distribution, they may fail to predict correct answers. In this work, we were concerned about this generalisation issue and thus analysed a broad set of models systematically and robustly over long stories. Related experiments were conducted based on the CLUTRR, which is a diagnostic benchmark suite that can analyse generalisation of natural language understanding (NLU) systems by training over small story graphs and testing on larger ones. In order to handle the multi-relational story graph, we consider two classes of neural models: "E-GNN", the graph-based models that can process graph-structured data and consider the edge attributes simultaneously; and "L-Graph", the sequence-based models which can process linearized version of the graphs. We performed an extensive empirical evaluation, and we found that the modified recurrent neural network yield surprisingly accurate results across every systematic generalisation tasks which outperform the modified graph neural network, while the latter produced more robust models.