Large Language Model Prediction Capabilities: Evidence from a Real-World
  Forecasting Tournament
Philipp Schoenegger Peter S. Park
Accurately predicting the future would be an important milestone in the
capabilities of artificial intelligence. However, research on the ability of
large language models to provide probabilistic predictions about future events
remains nascent. To empirically test this ability, we enrolled OpenAI's
state-of-the-art large language model, GPT-4, in a three-month forecasting
tournament hosted on the Metaculus platform. The tournament, running from July
to October 2023, attracted 843 participants and covered diverse topics
including Big Tech, U.S. politics, viral outbreaks, and the Ukraine conflict.
Focusing on binary forecasts, we show that GPT-4's probabilistic forecasts are
significantly less accurate than the median human-crowd forecasts. We find that
GPT-4's forecasts did not significantly differ from the no-information
forecasting strategy of assigning a 50% probability to every question. We
explore a potential explanation, that GPT-4 might be predisposed to predict
probabilities close to the midpoint of the scale, but our data do not support
this hypothesis. Overall, we find that GPT-4 significantly underperforms in
real-world predictive tasks compared to median human-crowd forecasts. A
potential explanation for this underperformance is that in real-world
forecasting tournaments, the true answers are genuinely unknown at the time of
prediction; unlike in other benchmark tasks like professional exams or time
series forecasting, where strong performance may at least partly be due to the
answers being memorized from the training data. This makes real-world
forecasting tournaments an ideal environment for testing the generalized
reasoning and prediction capabilities of artificial intelligence going forward.