Towards leveraging latent knowledge and Dialogue context for real-world conversational question answering
This work addresses the challenge of real-world conversational question answering where external knowledge is unavailable, but it is incremental as it builds on existing retrieval-reading methods with a summarization enhancement.
The paper tackles the problem of conversational question answering without external knowledge sources by proposing a neural Retrieval-Reading system that leverages latent knowledge from dialogue logs and uses a TFIDF-based summarizer to handle long contexts. The results show that this system generates significantly better answers and that the summarizer improves performance by providing more concise and less noisy context.
In many real-world scenarios, the absence of external knowledge source like Wikipedia restricts question answering systems to rely on latent internal knowledge in limited dialogue data. In addition, humans often seek answers by asking several questions for more comprehensive information. As the dialog becomes more extensive, machines are challenged to refer to previous conversation rounds to answer questions. In this work, we propose to leverage latent knowledge in existing conversation logs via a neural Retrieval-Reading system, enhanced with a TFIDF-based text summarizer refining lengthy conversational history to alleviate the long context issue. Our experiments show that our Retrieval-Reading system can exploit retrieved background knowledge to generate significantly better answers. The results also indicate that our context summarizer significantly helps both the retriever and the reader by introducing more concise and less noisy contextual information.