VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal
  Models
Zongjie Li Chaozheng Wang Chaowei Liu Pingchuan Ma Daoyuan Wu Shuai Wang Cuiyun Gao
With recent advancements in Large Multimodal Models (LMMs) across various
domains, a novel prompting method called visual referring prompting has
emerged, showing significant potential in enhancing human-computer interaction
within multimodal systems. This method offers a more natural and flexible
approach to human interaction with these systems compared to traditional text
descriptions or coordinates. However, the categorization of visual referring
prompting remains undefined, and its impact on the performance of LMMs has yet
to be formally examined. In this study, we conduct the first comprehensive
analysis of LMMs using a variety of visual referring prompting strategies. We
introduce a benchmark dataset called VRPTEST, comprising 3 different visual
tasks and 2,275 images, spanning diverse combinations of prompt strategies.
Using VRPTEST, we conduct a comprehensive evaluation of eight versions of
prominent open-source and proprietary foundation models, including two early
versions of GPT-4V. We develop an automated assessment framework based on
software metamorphic testing techniques to evaluate the accuracy of LMMs
without the need for human intervention or manual labeling. We find that the
current proprietary models generally outperform the open-source ones, showing
an average accuracy improvement of 22.70%; however, there is still potential
for improvement. Moreover, our quantitative analysis shows that the choice of
prompt strategy significantly affects the accuracy of LMMs, with variations
ranging from -17.5% to +7.3%. Further case studies indicate that an appropriate
visual referring prompting strategy can improve LMMs' understanding of context
and location information, while an unsuitable one might lead to answer
rejection. We also provide insights on minimizing the negative impact of visual
referring prompting on LMMs.