| |


Contact Greg Satell


Opinion 2,170 views

Are You Using Data To Analyze The Past Or To Create A New Future?

In Nate Silver’s Five Thirty Eight manifesto, he argues that the plural of anecdote is data.  It is through compiling and analyzing observations that we transform ordinary experiences into scientific conclusions.

Yet the concept of data is still relatively new.  According to Google’s Ngram, the use of the word only began to be heat up in the 1960’s.  So, although data is a term that gets thrown around a lot, it’s not something we’re really accustomed to using in a meaningful way.

That’s probably why many feel that data is in conflict with personal experience.  Yet in reality, data is most often used to codify experience.  Historical observations are aggregated and then analyzed to identify salient trends.  And that’s the real problem with how we use data.  History can be a trap.  It’s not the past that we need to worry about, but the future.

The Lure Of Historical Trends

My first real job out of college was on a trading desk, where I encountered the concept of “technical” market analysis.  Often known in the profession as “chartists,” practitioners scour historical data looking for patterns that they expect will give them clues to how things will play out in the future.

I was always amazed at the nearly religious fervor that many of the chartists had about their work, especially the Elliott wave theorists.  It was as if they were not merely performing analysis, but communing with nature and unlocking its secrets.  Chart analysis was a strange blend of the technical and the traditional.

I suspect that is part of what made the technique so popular.  Tradition has a romantic quality to it, as if we’re not merely forging our own path, but honoring the wisdom of those who came before us.  There has always been conflict between the medicine men, who put their faith in ageless insights and the visionaries who wish to break old molds.

The Poverty Of Historicism

In the 1930’s, Karl Popper began to speak out against excessive reverence of the past in a paper called The Poverty of Historicism, which was later published as a book.  He argued that, in fact, there was no reason to believe that future will look like what came before it.


World GDP per capita

image credit: Wikipedia


Take a look at the chart above and you can see what he means.  For most of history, little changed, but when the Industrial Revolution came along the world hit an inflection point.  It’s hard to argue that many trends in force before the shift remained valid afterwards.  Everything had to be called into question.

One of Popper’s students, George Soros, became one of the worlds richest men by identifying similar, albeit smaller shifts.  People like Henry Ford, Bill Gates and Steve Jobs defined entire new industries by breaking with established norms.  It is the visionaries we admire—not the trend followers—because they’re the ones who make an impact.

Why Big Data Matters

One of Popper’s key concepts was that any assertion, in order to be scientifically viable, has to be falsifiable. Unless it can be tested and proven wrong it was basically nonsense.  We admire great scientists like Darwin and Einstein not simply but because they had an idea, but because they stated it in a way that others could verify or disprove it.

For most of the 20th century, the testing process was deeply influenced by the methods of Ronald Fisher who set out rules for the design of experimentsconfidence intervals and statistical significance, among other things.  Underlying his ideas was an emphasis on controls.  Put good data in and you would get good answers out.

Yet for all of their charms, Fisher’s techniques were still essentially historical. You collected data over a period of time and determined mathematically whether they were meaningful or not.  You were still counting on the future looking like the past and, if it didn’t, you could find yourself wildly off the mark.

Post Your Comment

Please or sign up to comment.
