Is Graph Theory the Key to Understanding Big Data?

In the short course of its existence, Google has become a part of our daily lives. The search giant has become nearly unavoidable, due mostly to their ability to center services and products around the thing that they do best: indexing the Internet.

Google’s web crawling and PageRank algorithms revolutionized the way that people could search the Internet, delivering relevant information faster than ever before by cataloging the number and importance of web links.

These connections of links and sites come together to form a connection graph. Not a visual representation like we normally think of as a graph, but a kind of behind-the-scenes map that outlines how each page relates to all of the others.

PageRank uses these connections to judge how important or valuable a page might be. The Web pages that are hyperlinked by many third party sites are more likely to appear in a Google Search, as are pages linked by more authoritative information sources. Google search results are typically better and faster because their algorithms make the most of the connections between pages.

Applying these ideas of connections and relationships to other kinds of data can reveal the nature of the data itself, telling us what relates and what doesn’t, and what’s important and what isn’t.

When a Connection Becomes a Pattern Becomes a Graph

To see how data can be boiled down to reveal an answer, it’s important to understand the way that we have traditionally attempted to interact with data.  The underlying method for almost every process we have for finding answers is through search.

Invariably a search begins with the formulation of a question. The better we conceptualize what we know and how that relates to the data, the more likely our question will lead us to an answer. If you needed to find your keys, you might ask, “Where are my keys?” That’s not quite so descriptive as something like, “Are my keys on the counter?” The first question is probably too general to find your keys on its own, and the second is only a good question if your keys are, in fact, on the counter.

Querying a database works much the same way. Getting to the results you want requires constructing terms that relate to your data. There are an infinite number of queries you could run, but very few of them will lead you to the answer you are looking for.

This problem is exactly what makes data science so difficult and a good analyst such a valuable hire. The best data people are the ones that both know the data well and understand the technologies that are used to ask questions.

If the Web was your data set, then the search engine is your querying tool.

For decades, search engines have crawled the Web, indexing pages so that they could be found by search. The results of these searches varied, but by creating better search terms, users were able to change the results of their queries. Search engines worked to improve their products, but innovation in search was incremental until the early 2000’s.

By then Google’s PageRank had begun to catch on, organizing and ranking content by the connections that each link shared. Using graph connections, Google had quantified the connections of web pages to help users get to the right answer faster. The algorithm uses connections between pages to improve search results, but regardless of the search engine, more descriptive search terms will give the user a better result.

There is a relationship between your query and Google’s PageRank algorithm. Google has mapped, or graphed, the relationships between web pages to identify which pages are more relevant. Without this map of related pages and links, Google would require much better queries to reach an acceptable answer. Even with enhanced search techniques, modern data problems can make constructing the right query difficult or worse.

Understanding these relationships between data, whether it is Web pages, products purchased, features on a vehicle, words in a message, or symptoms, treatments, and outcomes from sick patients, is the first step to accepting that graphs will be the way we think about data in the future.

The data that we have today, and in often the ways we look at data, are already steeped in the theory of graphs. In the future, our ability to understand data with graphs will take us beyond searching for an answer. Creating and analyzing graphs will bring us to answers automatically. When we let data connect itself, meaning will begin to emerge automatically.

Dr. Roy Marsten, who has authored more than 30 papers on computational optimization and mathematics, is Chief Scientist at Emcien.


Views: 381

Tags: Analytics, Big Data



Join Innovation Insights

Comment by Edward Rotenberg on March 9, 2014 at 7:15pm

Sure it will be great: "When we let data connect itself, meaning will begin to emerge automatically." Automated algorithms do great job by fulfilling certain limited goals but they never take human completely out of decision loop. The final word is human. This is why visualization so important due to way human extracts quantitative information from visual environment. Yes, I fully agree that relations are the key but graphs per se are bad for human perception even in case of reasonable amount of nodes. I think we need some other way to leverage between multiple algorithms output and human consumption. You can find some visionary set of requirements in [1]. And you can not avoid to be interactive.




  • Add Videos
  • View All

© 2014   Created by Insights Site Creator.   Powered by

Badges  |  Report an Issue  |  Terms of Service