Approaching Human-Level Forecasting with Language Models
Danny Halawi Fred Zhang Chen Yueh-Han Jacob Steinhardt
Forecasting future events is important for policy and decision making. In
this work, we study whether language models (LMs) can forecast at the level of
competitive human forecasters. Towards this goal, we develop a
retrieval-augmented LM system designed to automatically search for relevant
information, generate forecasts, and aggregate predictions. To facilitate our
study, we collect a large dataset of questions from competitive forecasting
platforms. Under a test set published after the knowledge cut-offs of our LMs,
we evaluate the end-to-end performance of our system against the aggregates of
human forecasts. On average, the system nears the crowd aggregate of
competitive forecasters, and in some settings surpasses it. Our work suggests
that using LMs to forecast the future could provide accurate predictions at
scale and help to inform institutional decision making.