An Empirical Analysis of Diversity in Argument Summarization
This addresses the challenge of accommodating multiple perspectives in online societal discussions, which is incremental as it builds on existing argument summarization tasks like Key Point Analysis.
The paper tackled the problem of capturing diversity in argument summarization, finding that current approaches, including general-purpose LLMs and dedicated KPA models, struggle to represent minority arguments, handle diverse data sources, and align with subjective human annotations, with complementary strengths observed between methods.
Presenting high-level arguments is a crucial task for fostering participation in online societal discussions. Current argument summarization approaches miss an important facet of this task -- capturing diversity -- which is important for accommodating multiple perspectives. We introduce three aspects of diversity: those of opinions, annotators, and sources. We evaluate approaches to a popular argument summarization task called Key Point Analysis, which shows how these approaches struggle to (1) represent arguments shared by few people, (2) deal with data from various sources, and (3) align with subjectivity in human-provided annotations. We find that both general-purpose LLMs and dedicated KPA models exhibit this behavior, but have complementary strengths. Further, we observe that diversification of training data may ameliorate generalization. Addressing diversity in argument summarization requires a mix of strategies to deal with subjectivity.