e-QRAQ: A Multi-turn Reasoning Dataset and Simulator with Explanations
This addresses the problem of evaluating explainable reasoning in AI agents, though it is incremental as it builds on existing datasets and methods.
The paper introduces e-QRAQ, a dataset and simulator for testing agents' multi-turn reasoning with explanations, where agents read ambiguous stories, ask questions, and provide reasoning; they also train a neural architecture that shows a strong correlation between prediction quality and explanation quality.
In this paper we present a new dataset and user simulator e-QRAQ (explainable Query, Reason, and Answer Question) which tests an Agent's ability to read an ambiguous text; ask questions until it can answer a challenge question; and explain the reasoning behind its questions and answer. The User simulator provides the Agent with a short, ambiguous story and a challenge question about the story. The story is ambiguous because some of the entities have been replaced by variables. At each turn the Agent may ask for the value of a variable or try to answer the challenge question. In response the User simulator provides a natural language explanation of why the Agent's query or answer was useful in narrowing down the set of possible answers, or not. To demonstrate one potential application of the e-QRAQ dataset, we train a new neural architecture based on End-to-End Memory Networks to successfully generate both predictions and partial explanations of its current understanding of the problem. We observe a strong correlation between the quality of the prediction and explanation.