Inherent limitations of LLMs regarding spatial information
He Yan Xinyao Hu Xiangpeng Wan Chengyu Huang Kai Zou Shiqi Xu
Despite the significant advancements in natural language processing
capabilities demonstrated by large language models such as ChatGPT, their
proficiency in comprehending and processing spatial information, especially
within the domains of 2D and 3D route planning, remains notably underdeveloped.
This paper investigates the inherent limitations of ChatGPT and similar models
in spatial reasoning and navigation-related tasks, an area critical for
applications ranging from autonomous vehicle guidance to assistive technologies
for the visually impaired. In this paper, we introduce a novel evaluation
framework complemented by a baseline dataset, meticulously crafted for this
study. This dataset is structured around three key tasks: plotting spatial
points, planning routes in two-dimensional (2D) spaces, and devising pathways
in three-dimensional (3D) environments. We specifically developed this dataset
to assess the spatial reasoning abilities of ChatGPT. Our evaluation reveals
key insights into the model's capabilities and limitations in spatial
understanding.