Can LLMs Address Mental Health Questions? A Comparison with Human Therapists
This addresses the problem of limited mental health care access by evaluating LLMs as digital tools, though it is incremental in comparing existing models without new methods.
The study compared responses from large language models (LLMs) like ChatGPT, Gemini, and Llama to those from human therapists for mental health questions, finding that LLMs produced longer, more readable, and more positively toned responses rated as clearer and more supportive by users and therapists, but participants still preferred human support.
Limited access to mental health care has motivated the use of digital tools and conversational agents powered by large language models (LLMs), yet their quality and reception remain unclear. We present a study comparing therapist-written responses to those generated by ChatGPT, Gemini, and Llama for real patient questions. Text analysis showed that LLMs produced longer, more readable, and lexically richer responses with a more positive tone, while therapist responses were more often written in the first person. In a survey with 150 users and 23 licensed therapists, participants rated LLM responses as clearer, more respectful, and more supportive than therapist-written answers. Yet, both groups of participants expressed a stronger preference for human therapist support. These findings highlight the promise and limitations of LLMs in mental health, underscoring the need for designs that balance their communicative strengths with concerns of trust, privacy, and accountability.