VIM: Probing Multimodal Large Language Models for Visual Embedded
  Instruction Following
Yujie Lu Xiujun Li William Yang Wang Yejin Choi
We introduce VISUAL EMBEDDED INSTRUCTION (VIM), a new framework designed to
evaluate the visual instruction following capability of Multimodal Large
Language Models (MLLMs). As illustrated in Figure 2, VIM challenges the MLLMs
by embedding the instructions into the visual scenes, demanding strong visual
interpretative skills for instruction following. We adapt VIM to various
benchmarks, including VQAv2, MME, MM-Vet, and RefCOCO series, compose a VIM
bench, and probe diverse MLLMs across three distinct in-context learning
settings: Zero Shot, One Shot, and Pair Shot. We observe that there is a
significant performance disparity between the open-source MLLMs and GPT-4V,
implying that their proficiency in visual instruction comprehension is not up
to par. Our results highlight a promising direction for the enhancement of
MLLMs capabilities on instruction following. We aim VIM to serve as a useful
norm for advancing the state of the art and driving further progress in the
field.