Evaluating Prompting Strategies with MedGemma for Medical Order Extraction
This work addresses the problem of reducing clinical documentation burdens and ensuring patient safety for healthcare professionals, but it is incremental as it focuses on evaluating prompting strategies with an existing model.
The paper tackled medical order extraction from doctor-patient conversations using MedGemma, finding that a simple one-shot prompting method outperformed more complex frameworks like ReAct and agentic workflows on the validation set, with the one-shot approach achieving the highest performance.
The accurate extraction of medical orders from doctor-patient conversations is a critical task for reducing clinical documentation burdens and ensuring patient safety. This paper details our team submission to the MEDIQA-OE-2025 Shared Task. We investigate the performance of MedGemma, a new domain-specific open-source language model, for structured order extraction. We systematically evaluate three distinct prompting paradigms: a straightforward one-Shot approach, a reasoning-focused ReAct framework, and a multi-step agentic workflow. Our experiments reveal that while more complex frameworks like ReAct and agentic flows are powerful, the simpler one-shot prompting method achieved the highest performance on the official validation set. We posit that on manually annotated transcripts, complex reasoning chains can lead to "overthinking" and introduce noise, making a direct approach more robust and efficient. Our work provides valuable insights into selecting appropriate prompting strategies for clinical information extraction in varied data conditions.