ISQA: Informative Factuality Feedback for Scientific Summarization
This addresses the issue of factual accuracy in scientific summaries for researchers and practitioners, representing an incremental advancement in feedback-based refinement methods.
The paper tackled the problem of improving factuality in scientific summarization by proposing ISQA, a method that uses model-generated feedback with positive and negative information to iteratively refine summaries, resulting in significant improvements in factuality across various LLMs on multiple scientific datasets.
We propose Iterative Facuality Refining on Informative Scientific Question-Answering (ISQA) feedback\footnote{Code is available at \url{https://github.com/lizekai-richard/isqa}}, a method following human learning theories that employs model-generated feedback consisting of both positive and negative information. Through iterative refining of summaries, it probes for the underlying rationale of statements to enhance the factuality of scientific summarization. ISQA does this in a fine-grained manner by asking a summarization agent to reinforce validated statements in positive feedback and fix incorrect ones in negative feedback. Our findings demonstrate that the ISQA feedback mechanism significantly improves the factuality of various open-source LLMs on the summarization task, as evaluated across multiple scientific datasets.