Chinchilla Scaling: A replication attempt
Tamay Besiroglu Ege Erdil Matthew Barnett Josh You
Hoffmann et al. (2022) propose three methods for estimating a compute-optimal
scaling law. We attempt to replicate their third estimation procedure, which
involves fitting a parametric loss function to a reconstruction of data from
their plots. We find that the reported estimates are inconsistent with their
first two estimation methods, fail at fitting the extracted data, and report
implausibly narrow confidence intervals--intervals this narrow would require
over 600,000 experiments, while they likely only ran fewer than 500. In
contrast, our rederivation of the scaling law using the third approach yields
results that are compatible with the findings from the first two estimation
procedures described by Hoffmann et al.