Ruminating Reader: Reasoning with Gated Multi-Hop Attention
This work addresses the need for more accurate and reflective models in machine comprehension tasks, representing an incremental improvement over existing methods.
The paper tackles the problem of single-pass models in machine comprehension being unable to reflect on and correct answers by introducing Ruminating Reader, which adds a second pass of attention and a novel information fusion component to the BiDAF model, resulting in performance that matches or surpasses all other published systems on SQuAD.
To answer the question in machine comprehension (MC) task, the models need to establish the interaction between the question and the context. To tackle the problem that the single-pass model cannot reflect on and correct its answer, we present Ruminating Reader. Ruminating Reader adds a second pass of attention and a novel information fusion component to the Bi-Directional Attention Flow model (BiDAF). We propose novel layer structures that construct an query-aware context vector representation and fuse encoding representation with intermediate representation on top of BiDAF model. We show that a multi-hop attention mechanism can be applied to a bi-directional attention structure. In experiments on SQuAD, we find that the Reader outperforms the BiDAF baseline by a substantial margin, and matches or surpasses the performance of all other published systems.