MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models
This addresses the problem of evaluating psychiatric diagnostic capabilities in LLMs for mental health applications, but it is incremental as it builds on existing benchmark efforts.
The authors introduced MentalBench, a benchmark for evaluating psychiatric diagnostic decision-making in LLMs, using a psychiatrist-built knowledge graph to generate synthetic clinical cases, and found that while LLMs perform well on structured DSM-5 queries, they struggle with confidence calibration in overlapping disorders.
We introduce MentalBench, a benchmark for evaluating psychiatric diagnostic decision-making in large language models (LLMs). Existing mental health benchmarks largely rely on social media data, limiting their ability to assess DSM-grounded diagnostic judgments. At the core of MentalBench is MentalKG, a psychiatrist-built and validated knowledge graph encoding DSM-5 diagnostic criteria and differential diagnostic rules for 23 psychiatric disorders. Using MentalKG as a golden-standard logical backbone, we generate 24,750 synthetic clinical cases that systematically vary in information completeness and diagnostic complexity, enabling low-noise and interpretable evaluation. Our experiments show that while state-of-the-art LLMs perform well on structured queries probing DSM-5 knowledge, they struggle to calibrate confidence in diagnostic decision-making when distinguishing between clinically overlapping disorders. These findings reveal evaluation gaps not captured by existing benchmarks.