GPT4GEO: How a Language Model Sees the World's Geography
Jonathan Roberts Timo Lüddecke Sowmen Das Kai Han Samuel Albanie
Large language models (LLMs) have shown remarkable capabilities across a
broad range of tasks involving question answering and the generation of
coherent text and code. Comprehensively understanding the strengths and
weaknesses of LLMs is beneficial for safety, downstream applications and
improving performance. In this work, we investigate the degree to which GPT-4
has acquired factual geographic knowledge and is capable of using this
knowledge for interpretative reasoning, which is especially important for
applications that involve geographic data, such as geospatial analysis, supply
chain management, and disaster response. To this end, we design and conduct a
series of diverse experiments, starting from factual tasks such as location,
distance and elevation estimation to more complex questions such as generating
country outlines and travel networks, route finding under constraints and
supply chain analysis. We provide a broad characterisation of what GPT-4
(without plugins or Internet access) knows about the world, highlighting both
potentially surprising capabilities but also limitations.