Multi-Label Classification with Generative AI Models in Healthcare: A Case Study of Suicidality and Risk Factors
It addresses the critical need for early identification of complex suicidality risk factors in healthcare, offering a method to improve clinical interventions, though it is incremental as it builds on existing AI applications in this domain.
This study tackled the problem of identifying multiple co-occurring suicidality-related factors from psychiatric electronic health records by using generative large language models for multi-label classification, achieving top performance with finetuned GPT-3.5 at 0.94 partial match accuracy and 0.91 F1 score, and GPT-4.5 showing superior balanced performance across label sets.
Suicide remains a pressing global health crisis, with over 720,000 deaths annually and millions more affected by suicide ideation (SI) and suicide attempts (SA). Early identification of suicidality-related factors (SrFs), including SI, SA, exposure to suicide (ES), and non-suicidal self-injury (NSSI), is critical for timely intervention. While prior studies have applied AI to detect SrFs in clinical notes, most treat suicidality as a binary classification task, overlooking the complexity of cooccurring risk factors. This study explores the use of generative large language models (LLMs), specifically GPT-3.5 and GPT-4.5, for multi-label classification (MLC) of SrFs from psychiatric electronic health records (EHRs). We present a novel end to end generative MLC pipeline and introduce advanced evaluation methods, including label set level metrics and a multilabel confusion matrix for error analysis. Finetuned GPT-3.5 achieved top performance with 0.94 partial match accuracy and 0.91 F1 score, while GPT-4.5 with guided prompting showed superior performance across label sets, including rare or minority label sets, indicating a more balanced and robust performance. Our findings reveal systematic error patterns, such as the conflation of SI and SA, and highlight the models tendency toward cautious over labeling. This work not only demonstrates the feasibility of using generative AI for complex clinical classification tasks but also provides a blueprint for structuring unstructured EHR data to support large scale clinical research and evidence based medicine.