Heaps' Law in GPT-Neo Large Language Model Emulated Corpora
Uyen Lai Gurjit S. Randhawa Paul Sheridan
Heaps' law is an empirical relation in text analysis that predicts vocabulary
growth as a function of corpus size. While this law has been validated in
diverse human-authored text corpora, its applicability to large language model
generated text remains unexplored. This study addresses this gap, focusing on
the emulation of corpora using the suite of GPT-Neo large language models. To
conduct our investigation, we emulated corpora of PubMed abstracts using three
different parameter sizes of the GPT-Neo model. Our emulation strategy involved
using the initial five words of each PubMed abstract as a prompt and
instructing the model to expand the content up to the original abstract's
length. Our findings indicate that the generated corpora adhere to Heaps' law.
Interestingly, as the GPT-Neo model size grows, its generated vocabulary
increasingly adheres to Heaps' law as as observed in human-authored text. To
further improve the richness and authenticity of GPT-Neo outputs, future
iterations could emphasize enhancing model size or refining the model
architecture to curtail vocabulary repetition.