PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering
For clinicians and biomedical researchers, this work provides a practical, cost-controlled QA system that improves evidence grounding and trustworthiness over existing retrieval-augmented and self-reflection methods.
PubMed Reasoner introduces a three-stage biomedical QA agent that iteratively refines PubMed queries via self-critique and reflective retrieval, achieving 78.32% accuracy on PubMedQA (slightly surpassing human experts) and consistent gains on MMLU Clinical Knowledge, with LLM-as-judge evaluations favoring its responses across reasoning soundness, evidence grounding, clinical relevance, and trustworthiness.
Trustworthy biomedical question answering (QA) systems must not only provide accurate answers but also justify them with current, verifiable evidence. Retrieval-augmented approaches partially address this gap but lack mechanisms to iteratively refine poor queries, whereas self-reflection methods kick in only after full retrieval is completed. In this context, we introduce PubMed Reasoner, a biomedical QA agent composed of three stages: self-critic query refinement evaluates MeSH terms for coverage, alignment, and redundancy to enhance PubMed queries based on partial (metadata) retrieval; reflective retrieval processes articles in batches until sufficient evidence is gathered; and evidence-grounded response generation produces answers with explicit citations. PubMed Reasoner with a GPT-4o backbone achieves 78.32% accuracy on PubMedQA, slightly surpassing human experts, and showing consistent gains on MMLU Clinical Knowledge. Moreover, LLM-as-judge evaluations prefer our responses across: reasoning soundness, evidence grounding, clinical relevance, and trustworthiness. By orchestrating retrieval-first reasoning over authoritative sources, our approach provides practical assistance to clinicians and biomedical researchers while controlling compute and token costs.