ArgCMV: An Argument Summarization Benchmark for the LLM-era
This provides a more realistic benchmark for argument summarization research, addressing gaps in existing datasets, though it is incremental as it builds on prior work.
The authors tackled the limitations of existing datasets for argument key point extraction by creating ArgCMV, a new benchmark with 12K arguments from online debates across 3K topics, showing that current methods perform poorly on it.
Key point extraction is an important task in argument summarization which involves extracting high-level short summaries from arguments. Existing approaches for KP extraction have been mostly evaluated on the popular ArgKP21 dataset. In this paper, we highlight some of the major limitations of the ArgKP21 dataset and demonstrate the need for new benchmarks that are more representative of actual human conversations. Using SoTA large language models (LLMs), we curate a new argument key point extraction dataset called ArgCMV comprising of around 12K arguments from actual online human debates spread across over 3K topics. Our dataset exhibits higher complexity such as longer, co-referencing arguments, higher presence of subjective discourse units, and a larger range of topics over ArgKP21. We show that existing methods do not adapt well to ArgCMV and provide extensive benchmark results by experimenting with existing baselines and latest open source models. This work introduces a novel KP extraction dataset for long-context online discussions, setting the stage for the next generation of LLM-driven summarization research.