Fractal Patterns May Unravel the Intelligence in Next-Token Prediction
Ibrahim Alabdulmohsin Vinh Q. Tran Mostafa Dehghani
We study the fractal structure of language, aiming to provide a precise
formalism for quantifying properties that may have been previously suspected
but not formally shown. We establish that language is: (1) self-similar,
exhibiting complexities at all levels of granularity, with no particular
characteristic context length, and (2) long-range dependent (LRD), with a Hurst
parameter of approximately H=0.70. Based on these findings, we argue that
short-term patterns/dependencies in language, such as in paragraphs, mirror the
patterns/dependencies over larger scopes, like entire documents. This may shed
some light on how next-token prediction can lead to a comprehension of the
structure of text at multiple levels of granularity, from words and clauses to
broader contexts and intents. We also demonstrate that fractal parameters
improve upon perplexity-based bits-per-byte (BPB) in predicting downstream
performance. We hope these findings offer a fresh perspective on language and
the mechanisms underlying the success of LLMs.