Human vs. Machine: Language Models and Wargames
Max Lamparth Anthony Corso Jacob Ganz Oriana Skylar Mastro Jacquelyn Schneider Harold Trinkunas
Wargames have a long history in the development of military strategy and the
response of nations to threats or attacks. The advent of artificial
intelligence (AI) promises better decision-making and increased military
effectiveness. However, there is still debate about how AI systems, especially
large language models (LLMs), behave as compared to humans. To this end, we use
a wargame experiment with 107 national security expert human players designed
to look at crisis escalation in a fictional US-China scenario and compare human
players to LLM-simulated responses. We find considerable agreement in the LLM
and human responses but also significant quantitative and qualitative
differences between simulated and human players in the wargame, motivating
caution to policymakers before handing over autonomy or following AI-based
strategy recommendations.