Self-Adapting Large Visual-Language Models to Edge Devices across Visual
  Modalities
Kaiwen Cai Zhekai Duan Gaowen Liu Charles Fleming Chris Xiaoxuan Lu
Recent advancements in Vision-Language (VL) models have sparked interest in
their deployment on edge devices, yet challenges in handling diverse visual
modalities, manual annotation, and computational constraints remain. We
introduce EdgeVL, a novel framework that bridges this gap by seamlessly
integrating dual-modality knowledge distillation and quantization-aware
contrastive learning. This approach enables the adaptation of large VL models,
like CLIP, for efficient use with both RGB and non-RGB images on
resource-limited devices without the need for manual annotations. EdgeVL not
only transfers visual language alignment capabilities to compact models but
also maintains feature quality post-quantization, significantly enhancing
open-vocabulary classification performance across various visual modalities.
Our work represents the first systematic effort to adapt large VL models for
edge deployment, showcasing up to 15.4% accuracy improvements on multiple
datasets and up to 93-fold reduction in model size.