A Gold Standard Dataset and Evaluation Framework for Depression Detection and Explanation in Social Media using LLMs
This work addresses the need for better mental health monitoring tools by providing a dataset and framework for evaluating AI explanations in depression detection, though it is incremental as it builds on existing methods with new data.
The authors tackled the problem of depression detection from social media by creating a high-quality dataset of 1,017 posts with fine-grained depressive spans and symptom categories, and developed an evaluation framework to assess LLM-generated explanations, finding significant performance differences among models like GPT-4.1 and Claude 3.7 Sonnet.
Early detection of depression from online social media posts holds promise for providing timely mental health interventions. In this work, we present a high-quality, expert-annotated dataset of 1,017 social media posts labeled with depressive spans and mapped to 12 depression symptom categories. Unlike prior datasets that primarily offer coarse post-level labels \cite{cohan-etal-2018-smhd}, our dataset enables fine-grained evaluation of both model predictions and generated explanations. We develop an evaluation framework that leverages this clinically grounded dataset to assess the faithfulness and quality of natural language explanations generated by large language models (LLMs). Through carefully designed prompting strategies, including zero-shot and few-shot approaches with domain-adapted examples, we evaluate state-of-the-art proprietary LLMs including GPT-4.1, Gemini 2.5 Pro, and Claude 3.7 Sonnet. Our comprehensive empirical analysis reveals significant differences in how these models perform on clinical explanation tasks, with zero-shot and few-shot prompting. Our findings underscore the value of human expertise in guiding LLM behavior and offer a step toward safer, more transparent AI systems for psychological well-being.