ReadTwice: Reading Very Large Documents with Memories
This addresses the challenge of processing very large documents for question answering, with incremental improvements over prior methods.
The paper tackles the problem of assimilating information from large documents for knowledge-intensive tasks like question answering by proposing ReadTwice, a technique that reads text in segments and uses memory tables for a second pass, achieving state-of-the-art results on the NarrativeQA dataset.
Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections. We propose ReadTwice, a simple and effective technique that combines several strengths of prior approaches to model long-range dependencies with Transformers. The main idea is to read text in small segments, in parallel, summarizing each segment into a memory table to be used in a second read of the text. We show that the method outperforms models of comparable size on several question answering (QA) datasets and sets a new state of the art on the challenging NarrativeQA task, with questions about entire books. Source code and pre-trained checkpoints for ReadTwice can be found at https://goo.gle/research-readtwice.