Performance Assessment of ChatGPT vs Bard in Detecting Alzheimer's Dementia
This work evaluates LLMs for a specific medical diagnostic task, showing incremental performance improvements but not clinical readiness.
The study assessed ChatGPT-3.5, ChatGPT-4, and Bard in detecting Alzheimer's Dementia from speech text, finding that Bard had the highest recall for AD at 89% and F1 score of 71%, while GPT-4 had the highest true-negatives for CN at 56% and F1 score of 62%, but none met clinical standards.
Large language models (LLMs) find increasing applications in many fields. Here, three LLM chatbots (ChatGPT-3.5, ChatGPT-4 and Bard) are assessed - in their current form, as publicly available - for their ability to recognize Alzheimer's Dementia (AD) and Cognitively Normal (CN) individuals using textual input derived from spontaneous speech recordings. Zero-shot learning approach is used at two levels of independent queries, with the second query (chain-of-thought prompting) eliciting more detailed than the first. Each LLM chatbot's performance is evaluated on the prediction generated in terms of accuracy, sensitivity, specificity, precision and F1 score. LLM chatbots generated three-class outcome ("AD", "CN", or "Unsure"). When positively identifying AD, Bard produced highest true-positives (89% recall) and highest F1 score (71%), but tended to misidentify CN as AD, with high confidence (low "Unsure" rates); for positively identifying CN, GPT-4 resulted in the highest true-negatives at 56% and highest F1 score (62%), adopting a diplomatic stance (moderate "Unsure" rates). Overall, three LLM chatbots identify AD vs CN surpassing chance-levels but do not currently satisfy clinical application.