Scaling Efficient LLMs
B. N. Kausik
Trained LLMs are typically sparse in that most of the parameters are zero,
raising questions on efficiency. In response, we inquire into efficient LLMs,
i.e. those with the fewest parameters that achieve the desired accuracy on a
training corpus. Specifically, we compare theoretical and empirical estimates
for training loss at current scale to obtain upper and lower bounds on the
number of unique sequences in a natural training corpus as a function of its
size. Our result implies (1) to double the number of skills represented in a
training corpus, the corpus must scale roughly between three and five fold (2)
for efficient LLMs, the number of parameters $N$ and the size $D$ of a natural
training corpus scale as $N \sim D^{0.58}$ (3) if the number of parameters of
an LLM is smaller than the number of unique sequences in the training corpus,
scaling up can uncover emergent skills.