From Fuzzy Speech to Medical Insight: Benchmarking LLMs on Noisy Patient Narratives
This addresses the challenge of deploying LLMs in healthcare for real-world patient data, though it is incremental as it builds on existing benchmarks with a new dataset.
The authors tackled the problem of LLMs interpreting noisy patient narratives by creating a synthetic dataset with varying linguistic noise and evaluating models on it, finding that fine-tuned models achieved up to 85% accuracy in diagnosis tasks.
The widespread adoption of large language models (LLMs) in healthcare raises critical questions about their ability to interpret patient-generated narratives, which are often informal, ambiguous, and noisy. Existing benchmarks typically rely on clean, structured clinical text, offering limited insight into model performance under realistic conditions. In this work, we present a novel synthetic dataset designed to simulate patient self-descriptions characterized by varying levels of linguistic noise, fuzzy language, and layperson terminology. Our dataset comprises clinically consistent scenarios annotated with ground-truth diagnoses, spanning a spectrum of communication clarity to reflect diverse real-world reporting styles. Using this benchmark, we fine-tune and evaluate several state-of-the-art models (LLMs), including BERT-based and encoder-decoder T5 models. To support reproducibility and future research, we release the Noisy Diagnostic Benchmark (NDB), a structured dataset of noisy, synthetic patient descriptions designed to stress-test and compare the diagnostic capabilities of large language models (LLMs) under realistic linguistic conditions. We made the benchmark available for the community: https://github.com/lielsheri/PatientSignal