Evaluating Hierarchical Clinical Document Classification Using Reasoning-Based LLMs
This addresses the error-prone task of clinical document classification for healthcare professionals, but it is incremental as it tests existing models on a known problem.
This study evaluated large language models (LLMs) for classifying ICD-10 codes from hospital discharge summaries, finding that none achieved an F1 score above 57%, with reasoning-based models like Gemini 2.5 Pro performing best but still insufficient for full automation.
This study evaluates how well large language models (LLMs) can classify ICD-10 codes from hospital discharge summaries, a critical but error-prone task in healthcare. Using 1,500 summaries from the MIMIC-IV dataset and focusing on the 10 most frequent ICD-10 codes, the study tested 11 LLMs, including models with and without structured reasoning capabilities. Medical terms were extracted using a clinical NLP tool (cTAKES), and models were prompted in a consistent, coder-like format. None of the models achieved an F1 score above 57%, with performance dropping as code specificity increased. Reasoning-based models generally outperformed non-reasoning ones, with Gemini 2.5 Pro performing best overall. Some codes, such as those related to chronic heart disease, were classified more accurately than others. The findings suggest that while LLMs can assist human coders, they are not yet reliable enough for full automation. Future work should explore hybrid methods, domain-specific model training, and the use of structured clinical data.