LongRecall: A Structured Approach for Robust Recall Evaluation in Long-Form Text
This addresses the need for robust recall evaluation in domains like medicine and law, where omissions are critical, though it is an incremental improvement over existing methods.
The paper tackled the problem of evaluating recall in long-form text, where existing metrics fail due to lexical mismatches and hallucinations, by introducing LongRecall, a three-stage framework that decomposes answers into facts and verifies alignment through structured checks, achieving substantial improvements in recall accuracy on three QA benchmarks.
LongRecall. The completeness of machine-generated text, ensuring that it captures all relevant information, is crucial in domains such as medicine and law and in tasks like list-based question answering (QA), where omissions can have serious consequences. However, existing recall metrics often depend on lexical overlap, leading to errors with unsubstantiated entities and paraphrased answers, while LLM-as-a-Judge methods with long holistic prompts capture broader semantics but remain prone to misalignment and hallucinations without structured verification. We introduce LongRecall, a general three-stage recall evaluation framework that decomposes answers into self-contained facts, successively narrows plausible candidate matches through lexical and semantic filtering, and verifies their alignment through structured entailment checks. This design reduces false positives and false negatives while accommodating diverse phrasings and contextual variations, serving as a foundational building block for systematic recall assessment. We evaluate LongRecall on three challenging long-form QA benchmarks using both human annotations and LLM-based judges, demonstrating substantial improvements in recall accuracy over strong lexical and LLM-as-a-Judge baselines.