Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answers
For researchers evaluating generative models, WIMPE provides a more fine-grained and context-aware evaluation method that better aligns with human judgment.
The paper proposes a Weighted Importance Multi-Point Evaluation (WIMPE) framework for evaluating generative tasks with long-form answers, which factorizes reference answers into weighted context-bound scoring points and uses two metrics (WPA and PCP) to measure alignment and contradiction. Experiments on 10 tasks show WIMPE achieves higher correlations with human annotations.
Evaluating the quality of model responses remains challenging in generative tasks with long-form answers, as the expected answers usually contain multiple semantically distinct yet complementary factors that should be factorized for fine-grained assessment. Recent evaluation methods resort to relying on either task-level rubrics or question-aware checklists. However, they still 1) struggle to assess whether a response is genuinely grounded in provided contexts; 2) fail to capture the heterogeneous importance of different aspects of reference answers. Inspired by human examiners, we propose a Weighted Importance Multi-Point Evaluation (WIMPE) framework, which factorizes each reference answer into weighted context-bound scoring points. Two complementary metrics, namely Weighted Point-wise Alignment (WPA) and Point-wise Conflict Penalty (PCP), are designed to measure the alignment and contradiction between model responses and reference answers. Extensive experiments on 10 generative tasks demonstrate that WIMPE achieves higher correlations with human annotations.