Emerging Challenges in Personalized Medicine: Assessing Demographic Effects on Biomedical Question Answering Systems
This addresses fairness concerns in personalized medicine by highlighting frequent unjustified answer changes due to patient demographics, which is an incremental but critical issue for patient treatment.
The study investigated whether irrelevant demographic information affects biomedical question answering systems, finding that it changes up to 15% of answers in knowledge graph-grounded systems and up to 23% in text-based systems, including accuracy impacts.
State-of-the-art question answering (QA) models exhibit a variety of social biases (e.g., with respect to sex or race), generally explained by similar issues in their training data. However, what has been overlooked so far is that in the critical domain of biomedicine, any unjustified change in model output due to patient demographics is problematic: it results in the unfair treatment of patients. Selecting only questions on biomedical topics whose answers do not depend on ethnicity, sex, or sexual orientation, we ask the following research questions: (RQ1) Do the answers of QA models change when being provided with irrelevant demographic information? (RQ2) Does the answer of RQ1 differ between knowledge graph (KG)-grounded and text-based QA systems? We find that irrelevant demographic information change up to 15% of the answers of a KG-grounded system and up to 23% of the answers of a text-based system, including changes that affect accuracy. We conclude that unjustified answer changes caused by patient demographics are a frequent phenomenon, which raises fairness concerns and should be paid more attention to.