LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation
This work addresses a critical gap for researchers and practitioners in NLP by providing a more accurate evaluation tool for long document summarization, though it is incremental as it builds on existing metric approaches.
The paper tackles the problem of evaluating factual consistency in long document abstractive summarization, where existing metrics are inadequate due to token limits, and introduces LongDocFACTScore, a framework that outperforms state-of-the-art metrics by correlating better with human measures of factuality on a new dataset, LongSciVerify.
Maintaining factual consistency is a critical issue in abstractive text summarisation, however, it cannot be assessed by traditional automatic metrics used for evaluating text summarisation, such as ROUGE scoring. Recent efforts have been devoted to developing improved metrics for measuring factual consistency using pre-trained language models, but these metrics have restrictive token limits, and are therefore not suitable for evaluating long document text summarisation. Moreover, there is limited research and resources available for evaluating whether existing automatic evaluation metrics are fit for purpose when applied in long document settings. In this work, we evaluate the efficacy of automatic metrics for assessing the factual consistency of long document text summarisation. We create a human-annotated data set for evaluating automatic factuality metrics, LongSciVerify, which contains fine-grained factual consistency annotations for long document summaries from the scientific domain. We also propose a new evaluation framework, LongDocFACTScore, which is suitable for evaluating long document summarisation. This framework allows metrics to be efficiently extended to any length document and outperforms existing state-of-the-art metrics in its ability to correlate with human measures of factuality when used to evaluate long document summarisation data sets. We make our code and LongSciVerify data set publicly available: https://github.com/jbshp/LongDocFACTScore.