Distortions in Judged Spatial Relations in Large Language Models: The
  Dawn of Natural Language Geographic Data?
Nir Fulman Abdulkadir Memduhoğlu Alexander Zipf
We present a benchmark for assessing the capability of Large Language Models
(LLMs) to discern intercardinal directions between geographic locations and
apply it to three prominent LLMs: GPT-3.5, GPT-4, and Llama-2. This benchmark
specifically evaluates whether LLMs exhibit a hierarchical spatial bias similar
to humans, where judgments about individual locations' spatial relationships
are influenced by the perceived relationships of the larger groups that contain
them. To investigate this, we formulated 14 questions focusing on well-known
American cities. Seven questions were designed to challenge the LLMs with
scenarios potentially influenced by the orientation of larger geographical
units, such as states or countries, while the remaining seven targeted
locations less susceptible to such hierarchical categorization. Among the
tested models, GPT-4 exhibited superior performance with 55.3% accuracy,
followed by GPT-3.5 at 47.3%, and Llama-2 at 44.7%. The models showed
significantly reduced accuracy on tasks with suspected hierarchical bias. For
example, GPT-4's accuracy dropped to 32.9% on these tasks, compared to 85.7% on
others. Despite these inaccuracies, the models identified the nearest cardinal
direction in most cases, suggesting associative learning, embodying human-like
misconceptions. We discuss the potential of text-based data representing
geographic relationships directly to improve the spatial reasoning capabilities
of LLMs.