Evaluating Physician-AI Interaction for Cancer Management: Paving the Path towards Precision Oncology
For clinical decision support system designers and healthcare regulators, this study reveals that physicians exhibit automation bias toward ML recommendations, highlighting the need for safeguards before deployment in oncology.
Physicians over-relied on ML recommendations in a simulated cancer treatment task, showing automation bias even when tools for critical appraisal were provided. Concordant ML and RCT outputs increased confidence, while discordant results led most physicians to shift toward ML, often without reviewing model quality information.
As machine learning (ML)-based decision support tools proliferate in clinical practice, understanding how clinicians integrate personalized ML predictions alongside randomized controlled trial (RCT) evidence is critical. We designed a web-based clinical decision support system (CDSS) presenting survival and adverse event data from a simulated RCT and ML model across 12 synthetic multiple myeloma scenarios. In a within- subjects study with 32 physicians, we evaluated how clinicians synthesize competing evidence sources to make treatment decisions. When ML and RCT outputs were concordant, physicians reported greater confidence than with RCT data alone. When results were discordant, most physicians shifted toward the ML-supported treatment, often before reviewing any information about model training or validation, suggesting a tendency toward automation bias rather than algorithm avoidance. Despite reporting higher perceived reliability after viewing model quality disclosures, physicians were largely unable to describe the validation procedures they had reviewed. Taken together, these findings reveal that clinicians may over-rely on ML recommendations even when equipped with tools designed to support critical appraisal. We discuss implications for CDSS design, clinician training, and the institutional safeguards needed before ML-based systems are deployed in high-stakes oncology settings.