DM-Bench: Benchmarking LLMs for Personalized Decision Making in Diabetes Management
This addresses the need for reliable AI solutions in diabetes care by providing the first patient-facing benchmark for personalized decision-making, though it is incremental as it builds on existing health benchmarks by focusing on a specific domain.
The researchers tackled the lack of benchmarks for evaluating large language models (LLMs) in personalized diabetes management by creating DM-Bench, a comprehensive framework with 360,600 personalized questions across 7 task categories using data from 15,000 individuals. Their evaluation of 8 LLMs showed substantial variability across tasks and metrics, with no single model consistently outperforming others.
We present DM-Bench, the first benchmark designed to evaluate large language model (LLM) performance across real-world decision-making tasks faced by individuals managing diabetes in their daily lives. Unlike prior health benchmarks that are either generic, clinician-facing or focused on clinical tasks (e.g., diagnosis, triage), DM-Bench introduces a comprehensive evaluation framework tailored to the unique challenges of prototyping patient-facing AI solutions in diabetes, glucose management, metabolic health and related domains. Our benchmark encompasses 7 distinct task categories, reflecting the breadth of real-world questions individuals with diabetes ask, including basic glucose interpretation, educational queries, behavioral associations, advanced decision making and long term planning. Towards this end, we compile a rich dataset comprising one month of time-series data encompassing glucose traces and metrics from continuous glucose monitors (CGMs) and behavioral logs (e.g., eating and activity patterns) from 15,000 individuals across three different diabetes populations (type 1, type 2, pre-diabetes/general health and wellness). Using this data, we generate a total of 360,600 personalized, contextual questions across the 7 tasks. We evaluate model performance on these tasks across 5 metrics: accuracy, groundedness, safety, clarity and actionability. Our analysis of 8 recent LLMs reveals substantial variability across tasks and metrics; no single model consistently outperforms others across all dimensions. By establishing this benchmark, we aim to advance the reliability, safety, effectiveness and practical utility of AI solutions in diabetes care.