AI-Augmented Predictions: LLM Assistants Improve Human Forecasting
  Accuracy
Philipp Schoenegger Peter S. Park Ezra Karger Philip E. Tetlock
Large language models (LLMs) show impressive capabilities, matching and
sometimes exceeding human performance in many domains. This study explores the
potential of LLMs to augment judgement in forecasting tasks. We evaluated the
impact on forecasting accuracy of two GPT-4-Turbo assistants: one designed to
provide high-quality advice ('superforecasting'), and the other designed to be
overconfident and base-rate-neglecting. Participants (N = 991) had the option
to consult their assigned LLM assistant throughout the study, in contrast to a
control group that used a less advanced model (DaVinci-003) without direct
forecasting support. Our preregistered analyses reveal that LLM augmentation
significantly enhances forecasting accuracy by 23% across both types of
assistants, compared to the control group. This improvement occurs despite the
superforecasting assistant's higher accuracy in predictions, indicating the
augmentation's benefit is not solely due to model prediction accuracy.
Exploratory analyses showed a pronounced effect in one forecasting item,
without which we find that the superforecasting assistant increased accuracy by
43%, compared with 28% for the biased assistant. We further examine whether LLM
augmentation disproportionately benefits less skilled forecasters, degrades the
wisdom-of-the-crowd by reducing prediction diversity, or varies in
effectiveness with question difficulty. Our findings do not consistently
support these hypotheses. Our results suggest that access to an LLM assistant,
even a biased one, can be a helpful decision aid in cognitively demanding tasks
where the answer is not known at the time of interaction.