Can LLMs' Tuning Methods Work in Medical Multimodal Domain?
Jiawei Chen Yue Jiang Dingkang Yang Mingcheng Li Jinjie Wei Ziyun Qian Lihua Zhang
While large language models (LLMs) excel in world knowledge understanding,
adapting them to specific subfields requires precise adjustments. Due to the
model's vast scale, traditional global fine-tuning methods for large models can
be computationally expensive and impact generalization. To address this
challenge, a range of innovative Parameters-Efficient Fine-Tuning (PEFT)
methods have emerged and achieved remarkable success in both LLMs and Large
Vision-Language Models (LVLMs). In the medical domain, fine-tuning a medical
Vision-Language Pretrained (VLP) model is essential for adapting it to specific
tasks. Can the fine-tuning methods for large models be transferred to the
medical field to enhance transfer learning efficiency? In this paper, we delve
into the fine-tuning methods of LLMs and conduct extensive experiments to
investigate the impact of fine-tuning methods for large models on existing
multimodal models in the medical domain from the training data level and the
model structure level. We show the different impacts of fine-tuning methods for
large models on medical VLMs and develop the most efficient ways to fine-tune
medical VLP models. We hope this research can guide medical domain researchers
in optimizing VLMs' training costs, fostering the broader application of VLMs
in healthcare fields. Code and dataset will be released upon acceptance.