Dated Data: Tracing Knowledge Cutoffs in Large Language Models
Jeffrey Cheng Marc Marone Orion Weller Dawn Lawrie Daniel Khashabi Benjamin Van Durme
Released Large Language Models (LLMs) are often paired with a claimed
knowledge cutoff date, or the dates at which training data was gathered. Such
information is crucial for applications where the LLM must provide up to date
information. However, this statement only scratches the surface: do all
resources in the training data share the same knowledge cutoff date? Does the
model's demonstrated knowledge for these subsets closely align to their cutoff
dates? In this work, we define the notion of an effective cutoff. This is
distinct from the LLM designer reported cutoff and applies separately to
sub-resources and topics. We propose a simple approach to estimate effective
cutoffs on the resource-level temporal alignment of an LLM by probing across
versions of the data. Using this analysis, we find that effective cutoffs often
differ from reported cutoffs. To understand the root cause of this observation,
we conduct a direct large-scale analysis on open pre-training datasets. Our
analysis reveals two reasons for these inconsistencies: (1) temporal biases of
CommonCrawl data due to non-trivial amounts of old data in new dumps and (2)
complications in LLM deduplication schemes involving semantic duplicates and
lexical near-duplicates. Overall, our results show that knowledge cutoffs are
not as simple as they have seemed and that care must be taken both by LLM
dataset curators as well as practitioners who seek to use information from
these models.