Concert phones Flickr/Paul J Coles

By now, chances are you've heard the term "big data."


But the question for most people what? Why are techies and scientists so excited about this thing? After all, it's not like data didn't exist before, even in large quantities.

The short answer: big data is a new tech that can help us understand, and then predict, things like never before. Things like what we'll buy and when, when the prices of airline tickets and electronics are cheapest, which New York manholes are most likely to explode, or which tenement buildings most likely to catch fire.

Who wouldn't want to know stuff like that?


Correlations allow us to discover patterns and patterns allow us to predict the future, explains the authors of the book "Big Data: A Revolution that Will Transform How We Live, Work and Think," written by Oxford professor Viktor Mayer-Schonberger and big data journalist for the Economist, Kenneth Cukier.


Correlations are different from the classic method of making predictions: trying to find out "why" or "what caused something." That's known as "causation."

A correlation does not mean that one thing caused the other. For instance, earlier this year, MasterCard created its own big data division. After analyzing billions of records, it learned that when someone buys gas at around 4 p.m., the person will likely then spend $35-$50 on groceries or at a restaurant. The gas purchase didn't cause the food purchase.

"Predictions based on correlation lie at the heart of big data," the authors write.



  • Oren Etzioni, creator of one of the first search engines, MetaCrawler, scraped data from a travel website to build a tool called Farecast that predicted when an airfare price was likely to go down. Microsoft bought Farecast for $115 million in 2008. Today, it's part of Bing.