Why it’s way too early to dismiss big data’s economic impact


The New York Times published an article on Friday asking in its headline whether big data is a economic dud. Numerous quoted professors gave reasons why the analogy of data to oil doesn’t work, nor does the one of data to the electrical grid (and here I thought that analogy was about cloud computing).

All in all, it was kind of foolish. Here’s why.

1. Data is the new oil

Period. End of story. But we shouldn’t conflate data with big data. Data already has had significant effects on business and I can’t imagine anyone really taking issue with that statement. Whether or not it’s “big,” there’s no denying that all successful web companies and most successful companies of all types rely pretty heavily on data to make better decisions around marketing, operations or whatever.

Ask Google, Amazon, Walmart, Disney, GE, Target — and the list could go on for days — how important data is to them. The oil boom already happened, and it was great. Now, equal or not, it’s data that’s taking industry and commerce to the next level.

2. Big data is like oil sands

Big data is similar to plain, old data, but different in some important ways. Think about it like this: data is crude oil drilled from Saudi Arabian wells, while big data is the oil sands gunk that has lawmakers and businesspeople arguing so vehemently in the United States and Canada.

If regular data comes from sources such as customer information, purchase history and store sales numbers, big data comes from sources like social media, web searches, sensors and clickstreams. It takes a lot more work to turn that stuff into something usable. Right now, gains from big data at most companies (Google and Facebook aside), might be minimal at best.

This infographic from BloomReach helps illustrate how hard it complicated it can be to do big data right.


With that in mind, the question of whether big data is an economic dud is a little more fair. One might ask if all that effort is really worth the trouble to learn some potentially trivial insights about customer behavior. Maybe not, but that assumes big data is all about consumers’ personal data.

3. Big data and the web are not one and the same

A worldview of big data that’s limited to the web — which is the one the Times article espouses — is really missing a lot. Big data started on the web and a lot of technological innovation still happens there, but it’s making its way out. (Sometimes slowly — Sonic, a large fast-food chain, is just now getting started with big data, for example.) Ultimately, whatever effect the web has on the global economy might well be a drop in the bucket compared with how big data will affect other areas.

From a business perspective, you can already see what’s coming just by looking at Hadoop. It’s likely going to be the predominant big data platform going forward, and it’s constantly evolving to do new things. Companies big and small are using it (the vast majority of the Fortune 500 is at least doing proofs of concept, I’m told), some augmenting existing processes and others doing entirely new and innovative things they couldn’t otherwise do. The two biggest vendors focused solely on Hadoop — Cloudera and Hortonworks — are both talking about going public in the next year or two.

Looking broader, though, we can see even more promise on the horizon — the socioeconomic effect of which could be immense. There are companies like Planet Labs and Skybox using data to change the way we view Earth, and there are companies like Climate Corporation trying to rethink the economics of agriculture and technologies like real-time kinematics trying to make farming more efficient. Big data technology helped the United States track down Osama bin Laden and is (civil liberty concerns aside) helping governments identify potential terrorists and criminals all the time.

Rayid Ghani, once chief scientist for Obama for America, is pushing big data for social good at the University of Chicago and with a new startup called Edgeflip.


Data helped win a presidential election in 2012. I spent two hours last week (which I’ll detail in an upcoming post) talking about a litany of university projects trying to improve various social, municipal and health services using advances in data analysis. These include everything from aiding elections in Kenya to predicting hospital readmissions.

Some very smart people people think data analysis can do everything from predicting jet engine failures to curing cancer. Max Levchin thinks data can help women get pregnant. We have speakers at our upcoming Structure: Europe conference from both CERN and the European Space Agency. I’m guessing they’ll have a thing or two to say about how important big data technologies are to them.

Deep learning, arguably the holy grail for predictive analytics on big data sources like text, images and genomic data, is just making its way into the mainstream.

4. It’s way too early to call the game

I think a more accurate criticism of big data is probably to say it’s suffering from a case of being overhyped — vendors says their technologies do lots of things, but they don’t always spell out how much work that’s going to be. (Being overhyped, by the way, is not the same as being all hype.) As I’ve written before, anyone now getting upset that big data isn’t a perfect set of technologies and methodologies probably wasn’t paying much attention to what experts have been saying for years. But being overhyped now doesn’t mean big data can’t have a significant impact over time.


Source: Gartner Hype Cycle for Emerging Technologies

Yeah, it’s possible big data could turn out to be an economic dud, but there’s plenty of evidence that it’s just getting started in some significant areas. I’m gonna wait to see how all this plays out.

Feature image courtesy of Shutterstock user chuckstock.


Sam Moore

You make some great points, Derrick. We are still only scratching the surface, unlocking more and more potential for ‘big data.’ It’s not just the exponential growth in ‘1s’ and ‘0s’ that lead to spurring an economic growth, but it is the underlying correlation and causation between the tremendous amounts of information that contributes to more streamlined processes, better allocation of resources and more relevant ways to connect and monetize the relationship between business and customers. At BloomReach, we crunch terabytes of information to match an abstract concept like consumer intent to the right retailer content using a plethora of complex algorithms. And while, naturally, some consumers will make purchasing decisions over others, it’s the smart application of data – whether big or not – that leads to economic success. In just the last few years, we’ve seen tremendous career growth in the data field – newer classes of specialists that continue to innovate and create astounding methods that give competitive edges or generate entirely new avenues for revenue streams. It’s how you apply the gargantuan amount of data to work for each individualized business case that continues the economic evolution of big data.

Stephen Moffitt

One definition of ‘Big Data’ that has been put forward is Gartner’s:


This definition addresses what seems to be the main differences between old-fashioned ‘small’ data and the new ‘big’ variety: the amount of artifacts to manage, the speed at which they are created (e.g., the millions of clicks a day on the BBC website) and the fact that all these artifacts are created by different systems for different purposes with different structures, or no explicit structures at all.

The second part of the definition gets to Derrick’s ‘datafication’ point. There is no point in doing any of the information processing, and visualisation, if we do not have the implicit assumption that there is insight to be gathered from the activity.

At this point in time, there are indications that there is useful insight to be had in processing massive amounts of data, beyond targeted marketing and finding the left-footed fullback with the best pass completion rate over the last three years in the Premier League. I suspect that the first oil producers also did not know the extent of the possibilities of what they were making. The technology, in some senses, is the easiest part.

Alex Esterkin

I always like the Derrick’s metaphors and was hoping to enjoyably agree with the Derrick’s blog (as I usually do), but the following thesis stopped me from reading the rest of the blog: “If regular data comes from sources such as customer information, purchase history and store sales numbers, big data comes from sources like social media, web searches, sensors and clickstreams”. This statement cannot be used as an arguments, because it is profoundly incorrect.

Derrick Harris

I’m happy to hear your take on where big data comes from. Seriously.

Ideally, it all comes together in the end, but my point is that companies have been doing analytics for years. If it’s the 3 Vs that define “big” data, that data comes from different sources — certainly not limited to the ones I noted.

Comments are closed.