AI Content Moderation in Therapy Conversations
For developers and users of LLM-based therapy tools, this reveals a critical bottleneck where safety guardrails conflict with therapeutic needs.
The study audits three state-of-the-art AI content moderation systems (OpenAI, Meta, Google) on real therapy conversations, finding they frequently flag therapeutic content as undesirable, which limits LLMs' effectiveness as therapists.
Large language models (LLMs) are increasingly being used for emotional support. They are also being developed for formal therapy purposes. However, LLMs like ChaptGPT or Llama are often developed with content moderation guardrails that prevent them from discussing sensitive subjects with users for both liability and safety purposes, and this inability to broach these subjects may affect their capacity as therapists. In this study, we perform an algorithm audit on three state-of-the-art moderation systems (OpenAI's moderation endpoint, Meta's Llama Guard, and Google's Shield Gemma) to investigate the extent to which these systems flag the content of real-life therapy sessions as undesirable. Our results raise implications for the limitations that users and organizations may encounter when designing LLMs to play the part of a therapist.