"Big data" divides opinion

24 July 2012

NEW YORK: Exploiting the potential insights provided by "big data" could yield substantial opportunities, but also negative consequences, a survey of senior executives and experts has suggested.

Pew Internet, part of the Pew Research Center, and Elon University polled 1,021 industry leaders, analysts and commentators, some 53% of which agreed the rise of "big data" should "improve social, political and economic intelligence by 2020".

The specific tools anticipated to be available included "inferential software" assessing data patterns and project outcomes, algorithms establishing "advanced correlations", and systems for "nowcasting", or real-time forecasting.

"I'm a big believer in nowcasting", Hal Varian, chief economist at Google, the online company, said. "Nearly every large company has a real-time data warehouse and has more timely data on the economy than our government agencies."

Mark Watson, a senior engineer at Netflix, the video streaming platform, also suggested the greater rigor enabled by the vast quantity of statistical information on offer would be advantageous.

"We can expect huge benefits from these new possibilities to apply a scientific approach in areas, which previously were the subject only of speculation and theorising," he said.

Despite the broadly favourable view, 39% of those polled thought big data might "cause more problems than it solves" by 2020, such as by promoting "false confidence in our predictive powers".

"It's likely that there will be teething problems similar to the second, more cynical scenario, and there will definitely be attempts to manipulate the data to get it to support an opinion, ideology, or agenda," said Wesley George, principal engineer for Time Warner Cable's advanced technology group.

Among the other damaging connotations associated with this trend was the threat of "ignoring important outliers" and the minority, by simply serving the majority instead.

"The balance of positive to negative effect is very hard to judge," said Tracy Rolling, product user experience evangelist for Nokia. "It's like saying 'The Internet connects people and makes us more efficient. It is a huge positive for society,' and, 'The Internet is mostly used for porn."

Whatever the result, many contributors concurred that these shifts will occur over the longer term, both when considering actually extracting tangible knowledge, and for the wider implications to be felt.

"Data mining will be used more, but by 2020 it will still be in fairly limited ways for limited purposes, and won't have that much of an effect, though of course those marketing it will amplify the benefits. But it will probably be 2030 before it really gets powerful," Jonathan Grudin, principal researcher at Microsoft, said.

Data sourced from Elon University; additional content by Warc staff