REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction
This addresses a critical evaluation bottleneck for researchers and practitioners in natural language processing, particularly for generative models, by providing a more reliable framework, though it is incremental as it builds on existing matching techniques.
The paper tackles the problem that exact match evaluation underestimates generative event argument extraction models by ignoring valid variations, implicit arguments, and scattered arguments, and introduces REGen, a framework combining exact, relaxed, and LLM-based matching that shows an average performance gain of +23.93 F1 over exact match and 87.67% alignment with human assessments.
Event argument extraction identifies arguments for predefined event roles in text. Existing work evaluates this task with exact match (EM), where predicted arguments must align exactly with annotated spans. While suitable for span-based models, this approach falls short for large language models (LLMs), which often generate diverse yet semantically accurate arguments. EM severely underestimates performance by disregarding valid variations. Furthermore, EM evaluation fails to capture implicit arguments (unstated but inferable) and scattered arguments (distributed across a document). These limitations underscore the need for an evaluation framework that better captures models' actual performance. To bridge this gap, we introduce REGen, a Reliable Evaluation framework for Generative event argument extraction. REGen combines the strengths of exact, relaxed, and LLM-based matching to better align with human judgment. Experiments on six datasets show that REGen reveals an average performance gain of +23.93 F1 over EM, reflecting capabilities overlooked by prior evaluation. Human validation further confirms REGen's effectiveness, achieving 87.67% alignment with human assessments of argument correctness.