How predictable is language model benchmark performance?
David Owen
We investigate large language model performance across five orders of
magnitude of compute scaling in eleven recent model architectures. We show that
average benchmark performance, aggregating over many individual tasks and
evaluations as in the commonly-used BIG-Bench dataset, is decently predictable
as a function of training compute scale. Specifically, when extrapolating
BIG-Bench Hard performance across one order of magnitude in compute, we observe
average absolute errors of 6 percentage points (pp). By contrast, extrapolation
for individual BIG-Bench tasks across an order of magnitude in compute yields
higher average errors of 18pp. Nonetheless, individual task performance remains
significantly more predictable than chance. Overall, our work suggests compute
scaling provides a promising basis to forecast AI capabilities in diverse
benchmarks, though predicting performance in specific tasks poses challenges.