A Semantic QA-Based Approach for Text Summarization Evaluation
This addresses a long-standing challenge in NLP for researchers and practitioners by offering a novel evaluation approach, though it is incremental as it builds on existing QA and summarization techniques.
The paper tackles the problem of automatically evaluating text generation systems by proposing a semantic QA-based method to compare content differences between text passages, showing promising results on the 2007 DUC summarization corpus.
Many Natural Language Processing and Computational Linguistics applications involves the generation of new texts based on some existing texts, such as summarization, text simplification and machine translation. However, there has been a serious problem haunting these applications for decades, that is, how to automatically and accurately assess quality of these applications. In this paper, we will present some preliminary results on one especially useful and challenging problem in NLP system evaluation: how to pinpoint content differences of two text passages (especially for large pas-sages such as articles and books). Our idea is intuitive and very different from existing approaches. We treat one text passage as a small knowledge base, and ask it a large number of questions to exhaustively identify all content points in it. By comparing the correctly answered questions from two text passages, we will be able to compare their content precisely. The experiment using 2007 DUC summarization corpus clearly shows promising results.