When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models
This work addresses the information gap in transplant patient education materials, which is an incremental improvement for healthcare providers and patients by systematically measuring institutional variation.
The authors tackled the problem of quantifying heterogeneity in patient education materials for solid-organ transplantation across U.S. centers by introducing a framework using retrieval-augmented language models to compare answers from different handbooks. They found that 20.8% of non-absent pairwise comparisons showed clinically meaningful divergence, with coverage gaps in 96.2% of question-handbook pairs, particularly in reproductive health at 95.1% absence.
Patient education materials for solid-organ transplantation vary substantially across U.S. centers, yet no systematic method exists to quantify this heterogeneity at scale. We introduce a framework that grounds the same patient questions in different centers' handbooks using retrieval-augmented language models and compares the resulting answers using a five-label consistency taxonomy. Applied to 102 handbooks from 23 centers and 1,115 benchmark questions, the framework quantifies heterogeneity across four dimensions: question, topic, organ, and center. We find that 20.8% of non-absent pairwise comparisons exhibit clinically meaningful divergence, concentrated in condition monitoring and lifestyle topics. Coverage gaps are even more prominent: 96.2% of question-handbook pairs miss relevant content, with reproductive health at 95.1% absence. Center-level divergence profiles are stable and interpretable, where heterogeneity reflects systematic institutional differences, likely due to patient diversity. These findings expose an information gap in transplant patient education materials, with document-grounded medical question answering highlighting opportunities for content improvement.