Good at captioning, bad at counting: Benchmarking GPT-4V on Earth
  observation data
Chenhui Zhang Sherrie Wang
Large Vision-Language Models (VLMs) have demonstrated impressive performance
on complex tasks involving visual input with natural language instructions.
However, it remains unclear to what extent capabilities on natural images
transfer to Earth observation (EO) data, which are predominantly satellite and
aerial images less common in VLM training data. In this work, we propose a
comprehensive benchmark to gauge the progress of VLMs toward being useful tools
for EO data by assessing their abilities on scene understanding, localization
and counting, and change detection tasks. Motivated by real-world applications,
our benchmark includes scenarios like urban monitoring, disaster relief, land
use, and conservation. We discover that, although state-of-the-art VLMs like
GPT-4V possess extensive world knowledge that leads to strong performance on
open-ended tasks like location understanding and image captioning, their poor
spatial reasoning limits usefulness on object localization and counting tasks.
Our benchmark will be made publicly available at https://vleo.danielz.ch/ and
on Hugging Face at
https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70
for easy model evaluation.