February 22, 2016

Zipf's law is so simple -- and so strange

Here's a quick explanation of Zipf's Law:
Zipf's law in its simplest form, as formulated in the thirties by American linguist George Kingsley Zipf, states surprisingly that the most frequently occurring word in a text appears twice as often as the next most frequent word, three times more than the third most frequent one, four times more than the fourth most frequent one, and so on.
I mean, seriously. How can this possibly be true? And yet it is. Each one of us can write a book using whatever words occur to us, and yet our book -- and all other books -- will always comply with this law. It's as if we're being governed by something we are neither aware of, nor understand. I've known about this for a long time but every time I see it referenced, as I did today at phys.org, I'm shocked. It goes all the way down the line, too. The 96th most often-used word is used 96 times less frequently than the most frequently-used word. And on and on. How can this be?

The linked article is about researchers who recently applied Zipf's law to ancient texts, and found that they too comply with this law. This search was occasioned by the existence of Big Data in our technologically enhanced world. Prior to this, Zipf's law had only been tested on a certain number of texts. The researchers thought there was no reason to limit their investigation into the resilience of this law. So they looked into texts from all around the world, including ancient texts -- and they found that they all follow Zipf's law. In any language, in any age, Zipf's law holds true.

Each time I encounter Zipf's law, I am shocked anew. It's as if there's a whole 'nother level of rules that guide our lives -- rules we neither sense nor grasp, and yet we use them unfailingly. It's just amazing.

PS: When quantum computers are realized, I suspect we'll see lots of new correlations in the world around us. What we are aware of is so much less than what's really there. It's exciting and as I keep saying, very strange.


cm said...

Hey K. Until now, I never heard of Zipf's law. It's hard to believe something that seems so random turns out to be so structured. I learned something new today.

writenow said...

Hi c. I'm impressed that you found this post so quickly. Just put it up today. And let's face it, I haven't been blogging lately. The urge just came over me today, kinda like the Holy Ghost. Gasp! You don't think...Nah. There is no Holy Ghost.