Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning
This work addresses the performance and accuracy of on-device LLMs for medical applications, which is incremental as it benchmarks existing models on a specific dataset.
The study benchmarked on-device large language models for clinical reasoning using the AMEGA dataset, finding that compact general-purpose models like Phi-3 Mini balanced speed and accuracy well, while medically fine-tuned models such as Med42 and Aloe achieved the highest accuracy, with memory constraints being a greater challenge than processing power on older devices.
The deployment of Large Language Models (LLM) on mobile devices offers significant potential for medical applications, enhancing privacy, security, and cost-efficiency by eliminating reliance on cloud-based services and keeping sensitive health data local. However, the performance and accuracy of on-device LLMs in real-world medical contexts remain underexplored. In this study, we benchmark publicly available on-device LLMs using the AMEGA dataset, evaluating accuracy, computational efficiency, and thermal limitation across various mobile devices. Our results indicate that compact general-purpose models like Phi-3 Mini achieve a strong balance between speed and accuracy, while medically fine-tuned models such as Med42 and Aloe attain the highest accuracy. Notably, deploying LLMs on older devices remains feasible, with memory constraints posing a greater challenge than raw processing power. Our study underscores the potential of on-device LLMs for healthcare while emphasizing the need for more efficient inference and models tailored to real-world clinical reasoning.