Cloud-Device Collaborative Learning for Multimodal Large Language Models
Guanqun Wang Jiaming Liu Chenxuan Li Junpeng Ma Yuan Zhang Xinyu Wei Kevin Zhang Maurice Chong Ray Zhang Yijiang Liu Shanghang Zhang
The burgeoning field of Multimodal Large Language Models (MLLMs) has
exhibited remarkable performance in diverse tasks such as captioning,
commonsense reasoning, and visual scene understanding. However, the deployment
of these large-scale MLLMs on client devices is hindered by their extensive
model parameters, leading to a notable decline in generalization capabilities
when these models are compressed for device deployment. Addressing this
challenge, we introduce a Cloud-Device Collaborative Continual Adaptation
framework, designed to enhance the performance of compressed, device-deployed
MLLMs by leveraging the robust capabilities of cloud-based, larger-scale MLLMs.
Our framework is structured into three key components: a device-to-cloud uplink
for efficient data transmission, cloud-based knowledge adaptation, and an
optimized cloud-to-device downlink for model deployment. In the uplink phase,
we employ an Uncertainty-guided Token Sampling (UTS) strategy to effectively
filter out-of-distribution tokens, thereby reducing transmission costs and
improving training efficiency. On the cloud side, we propose Adapter-based
Knowledge Distillation (AKD) method to transfer refined knowledge from
large-scale to compressed, pocket-size MLLMs. Furthermore, we propose a Dynamic
Weight update Compression (DWC) strategy for the downlink, which adaptively
selects and quantizes updated weight parameters, enhancing transmission
efficiency and reducing the representational disparity between cloud and device
models. Extensive experiments on several multimodal benchmarks demonstrate the
superiority of our proposed framework over prior Knowledge Distillation and
device-cloud collaboration methods. Notably, we also validate the feasibility
of our approach to real-world experiments.