Reranking-based Generation for Unbiased Perspective Summarization
This work addresses the challenge of unbiased summarization in applications like political perspective summarization, representing an incremental advance in evaluation and method development.
The paper tackled the problem of generating unbiased perspective summaries by identifying reliable metrics for evaluating summary quality and showing that reranking-based methods with preference tuning improve performance, achieving strong results as measured by language model-based metrics.
Generating unbiased summaries in real-world settings such as political perspective summarization remains a crucial application of Large Language Models (LLMs). Yet, existing evaluation frameworks rely on traditional metrics for measuring key attributes such as coverage and faithfulness without verifying their applicability, and efforts to develop improved summarizers are still nascent. We address these gaps by (1) identifying reliable metrics for measuring perspective summary quality, and (2) investigating the efficacy of LLM-based methods beyond zero-shot inference. Namely, we build a test set for benchmarking metric reliability using human annotations and show that traditional metrics underperform compared to language model-based metrics, which prove to be strong evaluators. Using these metrics, we show that reranking-based methods yield strong results, and preference tuning with synthetically generated and reranking-labeled data further boosts performance. Our findings aim to contribute to the reliable evaluation and development of perspective summarization methods.