Big data is a hot topic. Everyone from world leaders, to CXOs, to analysts, to media and just about anyone connected to the technology industry is talking about the transformative powers of big data. It even made it onto the agenda of the 2012 annual meeting of the World Economic Forum in Davos. If the hype is to be believed, it has the power to transform businesses, governments and even society itself. It has the power to bring new insights into just about everything and drive a new era of intelligent understanding.

Yet if there's such potential here, why aren't we seeing more progress? There can be no question that some companies do understand the opportunities. One need only look at Google or Facebook to realise how these companies are effectively mining their data to drive new business opportunities and further monetize their value proposition. But these companies are rare exceptions. As I highlighted in my previous couple of blog posts, many companies are still very sceptical about moving from traditional business models and reluctant to embrace new opportunities.

However, resistance can only last so long, especially when we're talking about new revenue opportunities. Last month's World Economic Forum meeting saw people referring to data as a new class of economic asset and in many respects we're at the start of a new gold rush. One need only look at the job market to see proof of this. Take a look at job opportunities for data scientists. The results are staggering. There's a huge shortage of trained professionals. McKinsey & Company estimates this shortage to be almost 1.7 million big data professionals in the U.S. alone. It may be time to take your pan and start sifting.

Looking outside of staffing though, there are still two enormous obstacles. One is cultural and the second is infrastructure.

Firstly, if big data is to truly succeed we need to consider what data is being collected. Only this week headlines were made as the British government released plans to store details of every citizen’s phone calls, text messages, emails and website activity. I'm sure this act will meet with the same response as a Facebook privacy update. Yet it raises a key question, how do we feel about our data being collected and analysed? Can we overcome this barrier to see there are benefits? Can big data help us create, as Rick Smolan suggests, a human dashboard? Or will big data simply become another term for big brother?

Secondly, there can be no big data without big infrastructure. Data is useless if it sits in isolated silos. It needs to be harnessed, it needs to be collected on an enormous scale to be effectively analysed to the extent where it can offer insight and this is where we need a capable underlying network infrastructure. Whether this information is housed in the cloud or kept in your own data centres, it needs to be readily accessible, and this means we need to transport huge amounts of data quickly and securely.

What's incredible to note is the speed at which this data is growing. IDC suggests that data is doubling every two years. This is phenomenal growth and if we're to successfully transport this volume of data through the network, we need to continue driving forward with the optical reboot. A reboot based on the rebuilding of our networks on a core of 100G, OTNs and ROADMs and further expansion of fibre-based access and backhaul solutions. Something we’re only just at the start of doing.

What’s fascinating to note is how this data could spur a whole new industry. With Amazon's HPC-as-a-Service platform, we're seeing the democratization of supercomputing technology and this means big data can be accessed and analysed by a whole new subset of companies, groups and even individuals. It's tantalising to consider what breakthroughs this may result in. This will be especially true when the Internet of Things starts to mature and we have access to a whole other subset of data.

Are you involved in big data? Have you seen the impact it can have? What's the future here and what obstacles do we still need to overcome. I'd be interested to hear from you on this.