ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in
  Large Multimodal Models
Rohan Wadhawan Hritik Bansal Kai-Wei Chang Nanyun Peng
Recent advancements in AI have led to the development of large multimodal
models (LMMs) capable of processing complex tasks involving joint reasoning
over text and visual content in the image (e.g., navigating maps in public
places). This paper introduces ConTextual, a novel benchmark comprising
instructions designed explicitly to evaluate LMMs' ability to perform
context-sensitive text-rich visual reasoning. ConTextual emphasizes diverse
real-world scenarios (e.g., time-reading, navigation, shopping and more)
demanding a deeper understanding of the interactions between textual and visual
elements. Our findings reveal a significant performance gap of 30.8% between
the best-performing LMM, GPT-4V(ision), and human capabilities using human
evaluation indicating substantial room for improvement in context-sensitive
text-rich visual reasoning. Notably, while GPT-4V excelled in abstract
categories like meme and quote interpretation, its overall performance still
lagged behind humans. In addition to human evaluations, we also employed
automatic evaluation metrics using GPT-4, uncovering similar trends in
performance disparities. We also perform a fine-grained evaluation across
diverse visual contexts and provide qualitative analysis which provides a
robust framework for future advancements in the LMM design.
https://con-textual.github.io/