Autocorrelations Decay in Texts and Applicability Limits of Language
  Models
Nikolay Mikhaylovskiy Ilya Churilov
We show that the laws of autocorrelations decay in texts are closely related
to applicability limits of language models. Using distributional semantics we
empirically demonstrate that autocorrelations of words in texts decay according
to a power law. We show that distributional semantics provides coherent
autocorrelations decay exponents for texts translated to multiple languages.
The autocorrelations decay in generated texts is quantitatively and often
qualitatively different from the literary texts. We conclude that language
models exhibiting Markov behavior, including large autoregressive language
models, may have limitations when applied to long texts, whether analysis or
generation.