MedEdit: Model Editing for Medical Question Answering with External
  Knowledge Bases
Yucheng Shi Shaochen Xu Zhengliang Liu Tianming Liu Xiang Li Ninghao Liu
Large Language Models (LLMs), although powerful in general domains, often
perform poorly on domain-specific tasks like medical question answering (QA).
Moreover, they tend to function as "black-boxes," making it challenging to
modify their behavior. Addressing this, our study delves into model editing
utilizing in-context learning, aiming to improve LLM responses without the need
for fine-tuning or retraining. Specifically, we propose a comprehensive
retrieval strategy to extract medical facts from an external knowledge base,
and then we incorporate them into the query prompt for the LLM. Focusing on
medical QA using the MedQA-SMILE dataset, we evaluate the impact of different
retrieval models and the number of facts provided to the LLM. Notably, our
edited Vicuna model exhibited an accuracy improvement from 44.46% to 48.54%.
This work underscores the potential of model editing to enhance LLM
performance, offering a practical approach to mitigate the challenges of
black-box LLMs.