EchoQA: A Large Collection of Instruction Tuning Data for Echocardiogram Reports
This work addresses the need for better AI support in cardiology to reduce documentation burden and clinician burnout, though it is incremental as it builds on existing QA and LLM methods.
The authors tackled the problem of enhancing question-answering systems in cardiology by introducing EchoQA, a large dataset of 771,244 QA pairs from echocardiogram reports, and showed that fine-tuning large language models improves performance across various metrics.
We introduce a novel question-answering (QA) dataset using echocardiogram reports sourced from the Medical Information Mart for Intensive Care database. This dataset is specifically designed to enhance QA systems in cardiology, consisting of 771,244 QA pairs addressing a wide array of cardiac abnormalities and their severity. We compare large language models (LLMs), including open-source and biomedical-specific models for zero-shot evaluation, and closed-source models for zero-shot and three-shot evaluation. Our results show that fine-tuning LLMs improves performance across various QA metrics, validating the value of our dataset. Clinicians also qualitatively evaluate the best-performing model to assess the LLM responses for correctness. Further, we conduct fine-grained fairness audits to assess the bias-performance trade-off of LLMs across various social determinants of health. Our objective is to propel the field forward by establishing a benchmark for LLM AI agents aimed at supporting clinicians with cardiac differential diagnoses, thereby reducing the documentation burden that contributes to clinician burnout and enabling healthcare professionals to focus more on patient care.