Avoiding Short-Term Thinking In A World of Big Data

The promise of a world in which we collect massive amounts of data is that it will change our behavior for the better. But in a sea of data, how can we make sure that we’re not just reacting to the information in front of our face, but rather analyzing every possible input?

A few years ago, a group of economists led by Nobel prize winning Joseph Stiglitz tried to develop measurements of societal well-being beyond the standard metric of GDP, which only measures a narrow spectrum of economic activity. Beyond any of their specific recommendations, though, their report brought to light the history of GDP, a history that is incredibly instructive for the coming age of big data. The story? GDP was never meant to serve as a proxy for social well-being, but simply as a way to track economic output.

Over time, and with few other tangible metrics that give any sense of how society is doing, GDP has inadvertently become the default measurement that governments use to evaluate how much, or little, they’re doing to enhance well-being. As Stiglitz wrote at the time, "what we measure affects what we do." And so that measurement, of economic output, has had an outsized impact on how governments have functioned, at the expense of policies that might focus, instead, on improving health, happiness, and environmental quality.

Now this might seem like it isn’t connected to the idea of big data, where the promise seems to be that in contrast to past limits, we will be able to track, measure, and understand just about anything we’d want to know about our behavior as individuals and as a society. With the amount of information in the world forecast to increase by a multiple of 44 between 2009 and 2020, most concerns center around how we’ll keep from becoming overwhelmed by information.

Here’s one likely way we’ll cut through the deluge: We’ll have so much information that seems important at any given point that we’ll gravitate toward clear measurements—and have increasingly less tolerance for the ambiguous. For most of us, if we try something, and it doesn’t quickly demonstrate that it’s working, we’ll move on.

Now, this may not sound that bad. But it’s a bias that has the potential to increasingly push toward favoring the short-term over the long-term, with potentially disastrous consequences.

One of the worst, recent examples of short-term changes in data causing major problems comes from food. Between 2006 and 2008, global rice prices doubled, plunging hundreds of millions of people worldwide into hunger. Among the causes? Some local weather disruptions started pushing prices up (PDF), and seeing prices creeping upward, governments and consumers started buying and hoarding large supplies, creating a vicious cycle that ultimately helped push the number of people battling hunger over the one billion mark.

What we measure affects what we do, and in that case, the ability to detect a short-term price increase and react to it made the effects of that short-term challenge much more severe. Here in the U.S., Fortune 500 companies such as Blockbuster and Countrywide have fallen apart because of their focus on the short-term. Blockbuster in failing to realize that its users were moving rapidly toward streaming video and away from renting DVDs at brick and mortar stores, and Countrywide in making a short-term profit by giving out predatory loans, which would soon stop being paid.

This is a potential threat that will transcend a variety of sectors. There are any number of long-term benefits to education, but they’re ambiguous, hard to prove, and accumulate over an extended period of time. Big data may not help quantify that right away. Exercise improves a variety of health metrics over time—but in a world where everything is measured, it’s easy to imagine that most medical interventions will continue to focus on hospital treatments and medications that yield more immediate results, rather than on long-term, more nebulous investments (such as public parks where people can exercise) that create vague but substantial long-term benefits.

The opportunity, of course, in a world of big data, is that we’ll be increasingly able to not just measure short-term effects, but track and understand the effects of longer-term initiatives. The challenge will be to keep ourselves from acting for the short-term, in the face of so many measurements.

Add New Comment


  • david k waltz


    GDP is an economic measure rather than a social well-being measure. I think this actually serves as a point with respect to the topic. In a dearth of data environment it was the best proxy people found.

    With other types of data becoming available it will be possible to make new measures. Maybe positive / negative word usage on Twitter, as an example?


  • Paul B. Silverman

    Understanding The 'Unknown- Unknown' Information Drivers Helps Address 'Sea of Data' Issues and Creates Opportunities

    The article makes the point that "...in a sea of data, how can we make sure that we’re not just reacting to the information in front of our face, but rather analyzing every possible input."

    One solution to the problem, not mentioned in the article, is the need to develop new analytics to identify key drivers which create the data 'outcomes'. Predictive analytics enable us to identify these 'unknown-unknown' drivers that can only be found by analyzing data, looking for relationships and new rules that emerge developed by analyzing the data. Contrast this to today's 'deductive' approach using expert opinion and well-defined rules.

    This 'data-driven' analysis to create new rules is an inductive (rather than deductive 'expert opinion' based approach) and from my perspective holds great promise to radically change current business processes, improve productivity and improve our quality of life.

    This may sound bold, but as the former CEO of an early stage predictive analytics company and also looking at new opportunities in analytics, I see exciting potential here.

    Some possibilities:

    Look at manufacturing. If a “supplier’s supplier” has a problem, supply chain management ensures quick notification, before it impacts the assembly line. Predictive analytics engines 'raise the bar' here by analyzing historical performance and risk data, often real time, defining future risk and performance drivers, and enabling management to optimize performance and mitigate risk.

    Going beyond traditional data mining, these new predictive analytics tools analyze industry reports, government filings, trade press, and other sources to assess supplier “health,” pending regulations, and other “unstructured” data sources. Seamlessly integrating with other data, we can use these to more accurately gauge supplier and production line risk and improve performance.Driving new rules,  providing real time early warning signs that impact future supplier and business performance are the new management tools to harness 'the sea of data'.

    Look at health care, my primary focus, where PA techniques hold great promise to help our current health care system. Consider the benefits of these new capabilities which are only a small sample of what lies ahead here:

    •    Tracking  Medical Diagnoses, Treatments, Medications, Outcomes, Costs,Reimbursements, and Relationships

    ICD or International Classification of Disease Codes , classifies diseases on health records.CPT or Current Procedural Terminology codes developed by the AMA describe services provided by medical practitioners. Medicare employs a similar system, using 'HCPCS'. Tracking and examining
    relationships among these metrics, looking at patient data, identifying processes, and key cost and patient health drivers, you can develop 'best practices' to improve the health
    care process.

    •    Identifying Adverse Drug Analyses – assessing underlying drivers to more effectively identify “at risk” patients

    •    Optimizing clinical trials (candidate selection and monitoring) – predicting higher risk clinical trial candidates and assessing the key risk drivers

    •    Developing directional indicators to predict the underlying drivers for treatment of chronic disease to understand how medication protocols impact treatment plans and patient outcomes

    The new predictive analytic-based tools now emerging in all sectors are helping companies cope with the sea of data problem, and  “raising the bar” in how leading firms optimize business performance in today’s  dynamic global markets.

    Paul B. Silverman

    Author: Worm on a Chopstick : Understanding Today’s Entrepreneurial Age: Directions, Strategies, Management Perspectives http://paulbsilverman.com/book...

    Email:      paul@paulbsilverman.com
    blogs:       http://paulbsilverman.com/blog...
    Linked in:  Paul Silverman
    Twitter:     globalbizmentor