Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM
This work provides a performance reference point for LLMs in clinical diagnosis, showing that simple probabilistic baselines can account for substantial benchmark performance, which is incremental but useful for understanding and hybridizing AI methods.
The study investigated whether large language models (LLMs) rely on probabilistic reasoning for clinical diagnosis tasks by introducing a lightweight frequency-based method (FBPR) that matched LLM performance on MedQA benchmarks, with only slightly above random overlap in correct answers, indicating complementary strengths.
Large language models (LLMs) excel on multiple-choice clinical diagnosis benchmarks, yet it is unclear how much of this performance reflects underlying probabilistic reasoning. We study this through questions from MedQA, where the task is to select the most likely diagnosis. We introduce the Frequency-Based Probabilistic Ranker (FBPR), a lightweight method that scores options with a smoothed Naive Bayes over concept-diagnosis co-occurrence statistics from a large corpus. When co-occurrence statistics were sourced from the pretraining corpora for OLMo and Llama, FBPR achieves comparable performance to the corresponding LLMs pretrained on that same corpus. Direct LLM inference and FBPR largely get different questions correct, with an overlap only slightly above random chance, indicating complementary strengths of each method. These findings highlight the continued value of explicit probabilistic baselines: they provide a meaningful performance reference point and a complementary signal for potential hybridization. While the performance of LLMs seems to be driven by a mechanism other than simple frequency aggregation, we show that an approach similar to the historically grounded, low-complexity expert systems still accounts for a substantial portion of benchmark performance.