Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering
This addresses medical QA challenges for healthcare applications, but appears incremental as it builds on existing RAG and evaluation methods.
This study investigated LLM comprehension capabilities on long-context medical question answering, revealing insights into model size effects, memorization issues, and the benefits of reasoning models, while examining RAG strategies for improvements.
This study is the first to investigate LLM comprehension capabilities over long-context (LC) medical QA of clinical relevance. Our comprehensive assessment spans a range of content-inclusion settings based on their relevance, LLM models of varying capabilities and datasets across task formulations, revealing insights on model size effects, limitations, underlying memorization issues and the benefits of reasoning models. Importantly, we examine the effect of RAG on medical LC comprehension, uncover best settings in single versus multi-document reasoning datasets and showcase RAG strategies for improvements over LC. We shed light into some of the evaluation aspects using a multi-faceted approach. Our qualitative and error analyses address open questions on when RAG is beneficial over LC, revealing common failure cases.