Advancing Spatial Reasoning in Large Language Models: An In-Depth
  Evaluation and Enhancement Using the StepGame Benchmark
Fangjun Li David C. Hogg Anthony G. Cohn
Artificial intelligence (AI) has made remarkable progress across various
domains, with large language models like ChatGPT gaining substantial attention
for their human-like text-generation capabilities. Despite these achievements,
spatial reasoning remains a significant challenge for these models. Benchmarks
like StepGame evaluate AI spatial reasoning, where ChatGPT has shown
unsatisfactory performance. However, the presence of template errors in the
benchmark has an impact on the evaluation results. Thus there is potential for
ChatGPT to perform better if these template errors are addressed, leading to
more accurate assessments of its spatial reasoning capabilities. In this study,
we refine the StepGame benchmark, providing a more accurate dataset for model
evaluation. We analyze GPT's spatial reasoning performance on the rectified
benchmark, identifying proficiency in mapping natural language text to spatial
relations but limitations in multi-hop reasoning. We provide a flawless
solution to the benchmark by combining template-to-relation mapping with
logic-based reasoning. This combination demonstrates proficiency in performing
qualitative reasoning on StepGame without encountering any errors. We then
address the limitations of GPT models in spatial reasoning. We deploy
Chain-of-thought and Tree-of-thoughts prompting strategies, offering insights
into GPT's ``cognitive process", and achieving remarkable improvements in
accuracy. Our investigation not only sheds light on model deficiencies but also
proposes enhancements, contributing to the advancement of AI with more robust
spatial reasoning capabilities.