EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers
This work addresses the need for reliable evidence extraction in biomedical research, though it is incremental as it builds on existing annotation methods to create a new benchmark.
The authors tackled the problem of automatically extracting evidence relevant to hypotheses from biomedical papers by introducing EvidenceBench, a benchmark dataset created through a novel annotation pipeline, and found that current language models and retrieval systems perform significantly below expert levels on this task.
We study the task of automatically finding evidence relevant to hypotheses in biomedical papers. Finding relevant evidence is an important step when researchers investigate scientific hypotheses. We introduce EvidenceBench to measure models performance on this task, which is created by a novel pipeline that consists of hypothesis generation and sentence-by-sentence annotation of biomedical papers for relevant evidence, completely guided by and faithfully following existing human experts judgment. We demonstrate the pipeline's validity and accuracy with multiple sets of human-expert annotations. We evaluated a diverse set of language models and retrieval systems on the benchmark and found that model performances still fall significantly short of the expert level on this task. To show the scalability of our proposed pipeline, we create a larger EvidenceBench-100k with 107,461 fully annotated papers with hypotheses to facilitate model training and development. Both datasets are available at https://github.com/EvidenceBench/EvidenceBench