World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments
For researchers applying RL to clinical agents, the paper provides a taxonomy to predict RL learnability and prescribes a hybrid SFT+RL approach to overcome structural barriers.
The paper identifies a 41.7% silent-finish ceiling in MedAgentBench v1/v2 that makes inaction the dominant RL strategy, constructs MedAgentBench-v3 with an 8.9% ceiling, and shows that pure RL reaches only 18.2% pass@1 vs. 34.1% for rule-based SFT, with the 15.9 pp gap attributable to capability and format-knowledge barriers.
Clinical protocol-execution tasks -- checking a lab value, applying a threshold, placing a correctly structured FHIR order -- are natural candidates for RL from world feedback: once clinical SMEs encode decision logic into a verifier, that verifier grades unlimited rollouts without per-episode annotation. But applying RL requires a sound feedback channel and sufficient base capability. We audit MedAgentBench v1/v2, find a 41.7\% silent-finish ceiling that makes inaction the RL dominant strategy, and construct \textbf{MedAgentBench-v3 (MAB-v3)} (508 tasks, 8.9\% ceiling). Training Qwen3-8B exposes two structural barriers: a \emph{capability ceiling} (10/20 task types have 0\% base performance, zero gradient) and a \emph{format-knowledge barrier} (3/20 types require exact clinical codes undiscoverable by exploration). Pure RL reaches 18.2\% pass@1 vs.\ 34.1\% for rule-based SFT; the 15.9~pp gap is attributable entirely to these barriers. A decision/format-knowledge/lookup taxonomy predicts RL learnability and prescribes the fix: SFT to inject codes, RL to learn conditionals.