Enhancing Small Medical Learners with Privacy-preserving Contextual Prompting
This addresses data privacy concerns in healthcare by enhancing SLMs for medical applications, though it is incremental as it builds on existing prompting and privacy-preserving techniques.
The paper tackles the performance gap between large language models (LLMs) and small language models (SLMs) in medical tasks under privacy constraints by using LLMs to generate medical knowledge-intensive contexts from keywords, which boosts SLM decision-making. It achieves up to a 22.57% increase in absolute accuracy and sets new state-of-the-art results in two medical tasks.
Large language models (LLMs) demonstrate remarkable medical expertise, but data privacy concerns impede their direct use in healthcare environments. Although offering improved data privacy protection, domain-specific small language models (SLMs) often underperform LLMs, emphasizing the need for methods that reduce this performance gap while alleviating privacy concerns. In this paper, we present a simple yet effective method that harnesses LLMs' medical proficiency to boost SLM performance in medical tasks under privacy-restricted scenarios. Specifically, we mitigate patient privacy issues by extracting keywords from medical data and prompting the LLM to generate a medical knowledge-intensive context by simulating clinicians' thought processes. This context serves as additional input for SLMs, augmenting their decision-making capabilities. Our method significantly enhances performance in both few-shot and full training settings across three medical knowledge-intensive tasks, achieving up to a 22.57% increase in absolute accuracy compared to SLM fine-tuning without context, and sets new state-of-the-art results in two medical tasks within privacy-restricted scenarios. Further out-of-domain testing and experiments in two general domain datasets showcase its generalizability and broad applicability. Our code can be found at https://github.com/XZhang97666/PrivacyBoost-SLM.