Benchmarking and Adapting On-Device Large Language Models for Clinical Decision Support
This addresses privacy and resource constraints in deploying AI for clinical decision support, offering a practical solution for broader integration into healthcare, though it is incremental as it adapts existing models.
The study benchmarked on-device large language models (gpt-oss-20b and gpt-oss-120b) on clinical tasks, finding they performed comparably to or better than larger models like DeepSeek-R1 and o4-mini, and fine-tuning gpt-oss-20b improved its diagnostic accuracy to approach GPT-5 levels.
Large language models (LLMs) have rapidly advanced in clinical decision-making, yet the deployment of proprietary systems is hindered by privacy concerns and reliance on cloud-based infrastructure. Open-source alternatives allow local inference but often require large model sizes that limit their use in resource-constrained clinical settings. Here, we benchmark two on-device LLMs, gpt-oss-20b and gpt-oss-120b, across three representative clinical tasks: general disease diagnosis, specialty-specific (ophthalmology) diagnosis and management, and simulation of human expert grading and evaluation. We compare their performance with state-of-the-art proprietary models (GPT-5 and o4-mini) and a leading open-source model (DeepSeek-R1), and we further evaluate the adaptability of on-device systems by fine-tuning gpt-oss-20b on general diagnostic data. Across tasks, gpt-oss models achieve performance comparable to or exceeding DeepSeek-R1 and o4-mini despite being substantially smaller. In addition, fine-tuning remarkably improves the diagnostic accuracy of gpt-oss-20b, enabling it to approach the performance of GPT-5. These findings highlight the potential of on-device LLMs to deliver accurate, adaptable, and privacy-preserving clinical decision support, offering a practical pathway for broader integration of LLMs into routine clinical practice.