Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale
This addresses the problem of suicide screening for public health by assessing AI's potential role, but it is incremental as it applies existing LLMs to a new domain without introducing novel methods.
The study evaluated six large language models (LLMs) for automated suicide risk assessment using the Columbia-Suicide Severity Rating Scale, finding that Claude and GPT closely matched human annotations, while Mistral had the lowest ordinal prediction error, with misclassifications typically between adjacent severity levels.
Suicide prevention remains a critical public health challenge. While online platforms such as Reddit's r/SuicideWatch have historically provided spaces for individuals to express suicidal thoughts and seek community support, the advent of large language models (LLMs) introduces a new paradigm-where individuals may begin disclosing ideation to AI systems instead of humans. This study evaluates the capability of LLMs to perform automated suicide risk assessment using the Columbia-Suicide Severity Rating Scale (C-SSRS). We assess the zero-shot performance of six models-including Claude, GPT, Mistral, and LLaMA-in classifying posts across a 7-point severity scale (Levels 0-6). Results indicate that Claude and GPT closely align with human annotations, while Mistral achieves the lowest ordinal prediction error. Most models exhibit ordinal sensitivity, with misclassifications typically occurring between adjacent severity levels. We further analyze confusion patterns, misclassification sources, and ethical considerations, underscoring the importance of human oversight, transparency, and cautious deployment. Full code and supplementary materials are available at https://github.com/av9ash/llm_cssrs_code.