ViLaM: A Vision-Language Model with Enhanced Visual Grounding and
  Generalization Capability
Xiaoyu Yang Lijian Xu Hongsheng Li Shaoting Zhang
Vision-language models have revolutionized human-computer interaction and
shown significant progress in multi-modal tasks. However, applying these models
to complex visual tasks like medical image analysis remains challenging. In
this study, we propose ViLaM, a unified Vision-Language transformer model that
integrates instruction tuning predicated on a large language model. This
approach enables us to optimally utilize the knowledge and reasoning capacities
of large pre-trained language models for an array of tasks encompassing both
language and vision. We employ frozen pre-trained encoders to encode and align
both image and text features, enabling ViLaM to handle a variety of visual
tasks following textual instructions. Besides, we've designed cycle training
for referring expressions to address the need for high-quality, paired
referring expression datasets for training large models in terms of both
quantity and quality. We evaluated ViLaM's exceptional performance on public
general datasets and further confirmed its generalizability on medical
datasets. Importantly, we've observed the model's impressive zero-shot learning
ability, indicating the potential future application of ViLaM in the medical
field.