What's "up" with vision-language models? Investigating their struggle
  with spatial reasoning
Amita Kamath Jack Hessel Kai-Wei Chang
Recent vision-language (VL) models are powerful, but can they reliably
distinguish "right" from "left"? We curate three new corpora to quantify model
comprehension of such basic spatial relations. These tests isolate spatial
reasoning more precisely than existing datasets like VQAv2, e.g., our What'sUp
benchmark contains sets of photographs varying only the spatial relations of
objects, keeping their identity fixed (see Figure 1: models must comprehend not
only the usual case of a dog under a table, but also, the same dog on top of
the same table). We evaluate 18 VL models, finding that all perform poorly,
e.g., BLIP finetuned on VQAv2, which nears human parity on VQAv2, achieves 56%
accuracy on our benchmarks vs. humans at 99%. We conclude by studying causes of
this surprising behavior, finding: 1) that popular vision-language pretraining
corpora like LAION-2B contain little reliable data for learning spatial
relationships; and 2) that basic modeling interventions like up-weighting
preposition-containing instances or fine-tuning on our corpora are not
sufficient to address the challenges our benchmarks pose. We are hopeful that
these corpora will facilitate further research, and we release our data and
code at https://github.com/amitakamath/whatsup_vlms.