EyeGPT: Ophthalmic Assistant with Large Language Models
This work addresses the need for specialized AI assistants in ophthalmology to enhance patient experience and optimize services, representing an incremental improvement by adapting existing methods to a specific domain.
The authors tackled the problem of large language models lacking expert-level capability in medical tasks by developing EyeGPT, a specialized model for ophthalmology using role-playing, finetuning, and retrieval-augmented generation, which achieved comparable understandability, trustworthiness, and empathy to human ophthalmologists (all Ps>0.05).
Artificial intelligence (AI) has gained significant attention in healthcare consultation due to its potential to improve clinical workflow and enhance medical communication. However, owing to the complex nature of medical information, large language models (LLM) trained with general world knowledge might not possess the capability to tackle medical-related tasks at an expert level. Here, we introduce EyeGPT, a specialized LLM designed specifically for ophthalmology, using three optimization strategies including role-playing, finetuning, and retrieval-augmented generation. In particular, we proposed a comprehensive evaluation framework that encompasses a diverse dataset, covering various subspecialties of ophthalmology, different users, and diverse inquiry intents. Moreover, we considered multiple evaluation metrics, including accuracy, understandability, trustworthiness, empathy, and the proportion of hallucinations. By assessing the performance of different EyeGPT variants, we identify the most effective one, which exhibits comparable levels of understandability, trustworthiness, and empathy to human ophthalmologists (all Ps>0.05). Overall, ur study provides valuable insights for future research, facilitating comprehensive comparisons and evaluations of different strategies for developing specialized LLMs in ophthalmology. The potential benefits include enhancing the patient experience in eye care and optimizing ophthalmologists' services.