Graph Theory: Six Degrees of Separation Problem

Graph Theory: Six Degrees of Separation Problem




Note that in practice, people are connected through clusters of people, not randomly as in this model. Yet the simulator gives a pretty good idea of the process at play. In the above example, at level 5, you are already connected to 80% of all people.

Connections Simulator

The algorithm is described by this simple piece of Perl code: 




for ($level=1; $level<7; $level++) {
  for ($k=0; $k<$new[$level-1]; $k++) {
    for ($m=0; $m<$nfriends; $m++) { 
      $rand=(10232193*$rand + 3701101) % 54198451371;
      $friend= $rand % $n; 
      if ($a[$friend] == 0) {
  print “$level: $new[$level] – $npeople n”;

The array a stores the connections acquired at each level. The number of new connections at each level is equal to $new[$level], while the total reach (at each level) is $npeople. Note that depending on the parameters, you might not always be able to connect everyone to everyone. Just like in the real world, if some clusters of people are totally disconnected from the rest of the world, you won’t find a path between you and them.

The Need for a Good Random Number Generator

I have suspected for a long time that the function rand() in Perl provides a poor random generator. In this case, it fails miserably, being able to produce only 32,768 (that is, 2^15) different values, while we need 2,000,000. So I replaced it by a basic formula that fits the bill: This is what the computation of the variable $rand is for. Note the the operator % in the code, represents the modulo function. 

Application to Web Crawlers

When crawling the web, the best strategy is to proceed by successive layers as we did in our simulation: You start with (say) 1,000,000 seed URLs (easy to find using DMOZ or Quantcast) or even just 500 seed URLs, and at each iteration or level, you follow new links, making sure that you rarely if ever  revisit web pages that were previously crawled. It can be done on a laptop with multiple web crawlers running in parallel, if you are interested in only capturing 99.9% of the visible Internet. We did it, and here is the size of the data gathered at each level (compressed, in bytes):

The distribution is somewhat similar to the one shown in the first table, with a drop in the last level, and a rapid build-up during the first few levels.If you play with the simulator for a while (changing the parameters), you will occasionally stumble upon very similar distributions.

It took several months to gather all these links, using a 2-second timeout for each webpage crawl, downloading only the first 16 KB of each page, using a highly distributed architecture, and accessing only a few webpages per domain.

Some Mathematics

Watts and Strogatz showed that the average path length between two nodes in a random network is equal to log N / log K, where N = total nodes and K = acquaintances per node. Thus if N = 300,000,000 (90% of the US population) and K = 30 then Degrees of Separation = 5.7 and if N = 6,000,000,000 and K = 30 then Degrees of Separation = 6.6.

For related articles from the same author, click here or visit Follow me on Twitter at @GranvilleDSC or on LinkedIn.

DSC Resources

Popular Articles

Link: Graph Theory: Six Degrees of Separation Problem