Enhancing Small Medical Learners with Privacy-preserving Contextual
  Prompting
Xinlu Zhang Shiyang Li Xianjun Yang Chenxin Tian Yao Qin Linda Ruth Petzold
Large language models (LLMs) demonstrate remarkable medical expertise, but
data privacy concerns impede their direct use in healthcare environments.
Although offering improved data privacy protection, domain-specific small
language models (SLMs) often underperform LLMs, emphasizing the need for
methods that reduce this performance gap while alleviating privacy concerns. In
this paper, we present a simple yet effective method that harnesses LLMs'
medical proficiency to boost SLM performance in medical tasks under
privacy-restricted scenarios. Specifically, we mitigate patient privacy issues
by extracting keywords from medical data and prompting the LLM to generate a
medical knowledge-intensive context by simulating clinicians' thought
processes. This context serves as additional input for SLMs, augmenting their
decision-making capabilities. Our method significantly enhances performance in
both few-shot and full training settings across three medical
knowledge-intensive tasks, achieving up to a 22.57% increase in absolute
accuracy compared to SLM fine-tuning without context, and sets new
state-of-the-art results in two medical tasks within privacy-restricted
scenarios. Further out-of-domain testing and experiments in two general domain
datasets showcase its generalizability and broad applicability.