Coverage-based Fairness in Multi-document Summarization
This work addresses fairness issues in multi-document summarization for readers seeking comprehensive views, but it is incremental as it builds on existing fairness measures.
The paper tackled the problem of fairness in multi-document summarization by proposing new fairness measures, Equal Coverage and Coverage Parity, which account for document redundancy and corpus-level unfairness, and found that Claude3-sonnet was the fairest among thirteen evaluated LLMs, with most models overrepresenting different social attribute values.
Fairness in multi-document summarization (MDS) measures whether a system can generate a summary fairly representing information from documents with different social attribute values. Fairness in MDS is crucial since a fair summary can offer readers a comprehensive view. Previous works focus on quantifying summary-level fairness using Proportional Representation, a fairness measure based on Statistical Parity. However, Proportional Representation does not consider redundancy in input documents and overlooks corpus-level unfairness. In this work, we propose a new summary-level fairness measure, Equal Coverage, which is based on coverage of documents with different social attribute values and considers the redundancy within documents. To detect the corpus-level unfairness, we propose a new corpus-level measure, Coverage Parity. Our human evaluations show that our measures align more with our definition of fairness. Using our measures, we evaluate the fairness of thirteen different LLMs. We find that Claude3-sonnet is the fairest among all evaluated LLMs. We also find that almost all LLMs overrepresent different social attribute values. The code is available at https://github.com/leehaoyuan/coverage_fairness.