{"action":"created","comment":{"url":"https://api.github.com/repos/prometheus/docs/pulls/comments/33065044","id":33065044,"diff_hunk":"@@ -0,0 +1,122 @@\n+---\n+title: Monitoring Dreamhack - the World's Largest Digital Festival\n+created_at: 2015-06-20\n+kind: article\n+author_name: Christian Svensson (Dreamhack Network Team)\n+---\n+\n+*Disclaimer: This article is a guest post written by a Prometheus user*\n+\n+**If you are operating the network for 10 000's of demanding gamers you need to really know what is going on inside your network. Oh, and everything needs to be built from scratch in just 5 days.**\n+\n+If you never heard about Dreamhack before, here is the pitch:\n+Bring 20 000 people together and have the majority of them bring their own computer.\n+Mix in professional gaming (eSports), programming contests and live music concerts. The result is the world's largest festival dedicated solely to everything digital.\n+\n+To make such an event possible there needs to be a lot of infrastructure in place. Ordinary infrastructures of this size take months to build, but the crew at Dreamhack builds everything from scratch in just 5 days. This of course includes stuff like configuring network switches, but also building the electricity distribution, setting up stores for food and drinks and even building the actual tables.\n+\n+The team that builds and operates everything related to the network is officially called the Network team but we usually refers to ourselves as tech or dhtech. This post is going to focus on the work of dhtech and how we used Prometheus during Dreamhack Summer 2015 to try to kick our monitoring up another notch.\n+\n+## The Equipment\n+Turns out that to build a highly performant network for 10 000+ computers you need at least the same number of network ports. In our case these comes in the form of ~400 Cisco 2950 switches. We call these the Access switches. These are everywhere in venue where participants will be seated with their computers.\n+\n+[![Access switches](https://c1.staticflickr.com/9/8487/8206439882_4739d39a9c_c.jpg)](https://www.flickr.com/photos/dreamhack/8206439882)\n+<center>*Dutifully standing in line, the Access switches are ready to greet the Dreamhackers with high-speed connectivity.*</center>\n+\n+Obviously just connecting all these computers to a switch is not enough, that switch needs to be connected to the other switches as well. This is where the Distribution switches (or Dist switches) come into play. These are switches that take the hundreds of links from all access switches and aggregates them into more manageable 10 Gbit high-capacity fibre. The dist switches are then further aggregated into our Core where the traffic is routed to its destination. \n+\n+On top of all of this that we operate our own WiFi networks, DNS/DHCP servers and other infrastructure. When completed our Core looks something like below.\n+\n+[![The Dreamhack Network Core](https://c2.staticflickr.com/4/3951/18679671439_10ce7a8eb4_c.jpg)](https://www.flickr.com/photos/dreamhack/18679671439)\n+<center>*The Dreamhack Network Core*</center>\n+\n+[![Network Planing Map](http://i.imgur.com/ZCQa2Abl.png)](http://i.imgur.com/ZCQa2Ab.png)\n+<center>*The planning map for the Distribution and Core layers. The Core is clearly visible in \"Hall D\"*</center>\n+\n+All in all this is becoming a lengthy list of stuff to monitor, so let's get to the reason you're here: How do we make sure we know what's going on?\n+\n+## Introducing: dhmon\n+dhmon is the collective name of the systems that not only monitor the network, but also allows other teams to collect metrics on whatever they want.\n+\n+Since the network needs to be built in 5 days it's essential that the monitoring systems are easy to set up and keep in sync if we need to do last minute infrastructural changes (like adding or removing devices). When we start to build the network we need monitoring as soon as possible to be able to discover any problems with the equipment or other issues we hadn't foreseen.\n+\n+In the past we have tried to use a mix of commonly available software such as Cacti, SNMPc, Opsview among others. While these have worked they have focused on being closed systems and only provided the bare minimum. A few years back a few people from the team said \"Enough, we can do better ourselves!\" and started writing a custom monitoring solution.\n+\n+At the time the options were limited. Over the years the system went from using Graphite (scalability issues), custom Cassandra store (high complexity), InfluxDB (immature software) to finally land on using Prometheus. I first learned about Prometheus back in 2014 when I met Julius Voltz and I had been eager to try using it ever since. This summer we finally replaced the custom InfluxDB-based metric store that we had written with Prometheus. Spoiler: We're not going back.\n+\n+## The Architecture\n+The monitoring solution consists of three layers: collection, storage, presentation. Our most critical collectors are snmpcollector (SNMP) and ipplan-pinger (ICMP), closely followed by dhcpinfo (DHCP lease stats). We also have some scripts that dump stats about other systems into node_exporter's textfile collector.\n+\n+[![dhmon Architecture](http://i.imgur.com/6gN3MRp.png)](http://i.imgur.com/6gN3MRp.png)\n+<center>*The current architecture plan of dhmon as of Summer 2015*</center>\n+\n+As for storage we use Prometheus as a central timeseries storage and querying engine but we also use Redis and memcache to export snapshot views of binary information that we collect but cannot store in Prometheus in any sensible way or when we need to access very fresh data.\n+\n+One such case is in our presentation layer. We use our dhmap web application to get an overview of the overall health of the access switches. In order to be effective at resolving errors we need a latency of ~10 seconds from data collection to presentation. Our goal is to have fixed the problem before the customer notices, or at least before he/she has walked over to the support people to report an issue. For this reason, we have been using memcache since the beginning to access the latest snapshot of the network.","path":"content/blog/2015-06-20-monitoring-dreamhack.md","position":55,"original_position":55,"commit_id":"e760566ec9572f5905ce4815f6437ad0a557f053","original_commit_id":"e760566ec9572f5905ce4815f6437ad0a557f053","user":{"login":"fabxc","id":4948210,"avatar_url":"https://avatars.githubusercontent.com/u/4948210?v=3","gravatar_id":"","url":"https://api.github.com/users/fabxc","html_url":"https://github.com/fabxc","followers_url":"https://api.github.com/users/fabxc/followers","following_url":"https://api.github.com/users/fabxc/following{/other_user}","gists_url":"https://api.github.com/users/fabxc/gists{/gist_id}","starred_url":"https://api.github.com/users/fabxc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fabxc/subscriptions","organizations_url":"https://api.github.com/users/fabxc/orgs","repos_url":"https://api.github.com/users/fabxc/repos","events_url":"https://api.github.com/users/fabxc/events{/privacy}","received_events_url":"https://api.github.com/users/fabxc/received_events","type":"User","site_admin":false},"body":"If we are referring to the caching server, it's correct name is memchache*d*.","created_at":"2015-06-23T17:05:18Z","updated_at":"2015-06-23T17:05:18Z","html_url":"https://github.com/prometheus/docs/pull/123#discussion_r33065044","pull_request_url":"https://api.github.com/repos/prometheus/docs/pulls/123","_links":{"self":{"href":"https://api.github.com/repos/prometheus/docs/pulls/comments/33065044"},"html":{"href":"https://github.com/prometheus/docs/pull/123#discussion_r33065044"},"pull_request":{"href":"https://api.github.com/repos/prometheus/docs/pulls/123"}}},"pull_request":{"url":"https://api.github.com/repos/prometheus/docs/pulls/123","id":38358191,"html_url":"https://github.com/prometheus/docs/pull/123","diff_url":"https://github.com/prometheus/docs/pull/123.diff","patch_url":"https://github.com/prometheus/docs/pull/123.patch","issue_url":"https://api.github.com/repos/prometheus/docs/issues/123","number":123,"state":"open","locked":false,"title":"Guest Post: Monitoring Dreamhack - the World's Largest Digital Festival","user":{"login":"bluecmd","id":149442,"avatar_url":"https://avatars.githubusercontent.com/u/149442?v=3","gravatar_id":"","url":"https://api.github.com/users/bluecmd","html_url":"https://github.com/bluecmd","followers_url":"https://api.github.com/users/bluecmd/followers","following_url":"https://api.github.com/users/bluecmd/following{/other_user}","gists_url":"https://api.github.com/users/bluecmd/gists{/gist_id}","starred_url":"https://api.github.com/users/bluecmd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bluecmd/subscriptions","organizations_url":"https://api.github.com/users/bluecmd/orgs","repos_url":"https://api.github.com/users/bluecmd/repos","events_url":"https://api.github.com/users/bluecmd/events{/privacy}","received_events_url":"https://api.github.com/users/bluecmd/received_events","type":"User","site_admin":false},"body":"See commits for the guest story of how we used Prometheus to monitor Dreamhack.\r\n\r\nWhen I send this PR I still have one image to replace, but other than that: RFC.\r\n\r\nPing @juliusv.","created_at":"2015-06-23T16:50:44Z","updated_at":"2015-06-23T17:05:18Z","closed_at":null,"merged_at":null,"merge_commit_sha":"52e7307fb646055c1ece23fe0f9cd126e6e35f9e","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/prometheus/docs/pulls/123/commits","review_comments_url":"https://api.github.com/repos/prometheus/docs/pulls/123/comments","review_comment_url":"https://api.github.com/repos/prometheus/docs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/prometheus/docs/issues/123/comments","statuses_url":"https://api.github.com/repos/prometheus/docs/statuses/e760566ec9572f5905ce4815f6437ad0a557f053","head":{"label":"bluecmd:master","ref":"master","sha":"e760566ec9572f5905ce4815f6437ad0a557f053","user":{"login":"bluecmd","id":149442,"avatar_url":"https://avatars.githubusercontent.com/u/149442?v=3","gravatar_id":"","url":"https://api.github.com/users/bluecmd","html_url":"https://github.com/bluecmd","followers_url":"https://api.github.com/users/bluecmd/followers","following_url":"https://api.github.com/users/bluecmd/following{/other_user}","gists_url":"https://api.github.com/users/bluecmd/gists{/gist_id}","starred_url":"https://api.github.com/users/bluecmd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bluecmd/subscriptions","organizations_url":"https://api.github.com/users/bluecmd/orgs","repos_url":"https://api.github.com/users/bluecmd/repos","events_url":"https://api.github.com/users/bluecmd/events{/privacy}","received_events_url":"https://api.github.com/users/bluecmd/received_events","type":"User","site_admin":false},"repo":{"id":37773692,"name":"prometheus-docs","full_name":"bluecmd/prometheus-docs","owner":{"login":"bluecmd","id":149442,"avatar_url":"https://avatars.githubusercontent.com/u/149442?v=3","gravatar_id":"","url":"https://api.github.com/users/bluecmd","html_url":"https://github.com/bluecmd","followers_url":"https://api.github.com/users/bluecmd/followers","following_url":"https://api.github.com/users/bluecmd/following{/other_user}","gists_url":"https://api.github.com/users/bluecmd/gists{/gist_id}","starred_url":"https://api.github.com/users/bluecmd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bluecmd/subscriptions","organizations_url":"https://api.github.com/users/bluecmd/orgs","repos_url":"https://api.github.com/users/bluecmd/repos","events_url":"https://api.github.com/users/bluecmd/events{/privacy}","received_events_url":"https://api.github.com/users/bluecmd/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/bluecmd/prometheus-docs","description":"Prometheus documentation: content and static site generator","fork":true,"url":"https://api.github.com/repos/bluecmd/prometheus-docs","forks_url":"https://api.github.com/repos/bluecmd/prometheus-docs/forks","keys_url":"https://api.github.com/repos/bluecmd/prometheus-docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bluecmd/prometheus-docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bluecmd/prometheus-docs/teams","hooks_url":"https://api.github.com/repos/bluecmd/prometheus-docs/hooks","issue_events_url":"https://api.github.com/repos/bluecmd/prometheus-docs/issues/events{/number}","events_url":"https://api.github.com/repos/bluecmd/prometheus-docs/events","assignees_url":"https://api.github.com/repos/bluecmd/prometheus-docs/assignees{/user}","branches_url":"https://api.github.com/repos/bluecmd/prometheus-docs/branches{/branch}","tags_url":"https://api.github.com/repos/bluecmd/prometheus-docs/tags","blobs_url":"https://api.github.com/repos/bluecmd/prometheus-docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bluecmd/prometheus-docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bluecmd/prometheus-docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/bluecmd/prometheus-docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bluecmd/prometheus-docs/statuses/{sha}","languages_url":"https://api.github.com/repos/bluecmd/prometheus-docs/languages","stargazers_url":"https://api.github.com/repos/bluecmd/prometheus-docs/stargazers","contributors_url":"https://api.github.com/repos/bluecmd/prometheus-docs/contributors","subscribers_url":"https://api.github.com/repos/bluecmd/prometheus-docs/subscribers","subscription_url":"https://api.github.com/repos/bluecmd/prometheus-docs/subscription","commits_url":"https://api.github.com/repos/bluecmd/prometheus-docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/bluecmd/prometheus-docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/bluecmd/prometheus-docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/bluecmd/prometheus-docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/bluecmd/prometheus-docs/contents/{+path}","compare_url":"https://api.github.com/repos/bluecmd/prometheus-docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bluecmd/prometheus-docs/merges","archive_url":"https://api.github.com/repos/bluecmd/prometheus-docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bluecmd/prometheus-docs/downloads","issues_url":"https://api.github.com/repos/bluecmd/prometheus-docs/issues{/number}","pulls_url":"https://api.github.com/repos/bluecmd/prometheus-docs/pulls{/number}","milestones_url":"https://api.github.com/repos/bluecmd/prometheus-docs/milestones{/number}","notifications_url":"https://api.github.com/repos/bluecmd/prometheus-docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bluecmd/prometheus-docs/labels{/name}","releases_url":"https://api.github.com/repos/bluecmd/prometheus-docs/releases{/id}","created_at":"2015-06-20T15:03:04Z","updated_at":"2015-06-20T15:03:16Z","pushed_at":"2015-06-23T16:47:52Z","git_url":"git://github.com/bluecmd/prometheus-docs.git","ssh_url":"git@github.com:bluecmd/prometheus-docs.git","clone_url":"https://github.com/bluecmd/prometheus-docs.git","svn_url":"https://github.com/bluecmd/prometheus-docs","homepage":"http://prometheus.io","size":2121,"stargazers_count":0,"watchers_count":0,"language":"CSS","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"prometheus:master","ref":"master","sha":"fca980fbea1e98445b291c284dfd65d8a18a7ab8","user":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"repo":{"id":28233924,"name":"docs","full_name":"prometheus/docs","owner":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/prometheus/docs","description":"Prometheus documentation: content and static site generator","fork":false,"url":"https://api.github.com/repos/prometheus/docs","forks_url":"https://api.github.com/repos/prometheus/docs/forks","keys_url":"https://api.github.com/repos/prometheus/docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/prometheus/docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/prometheus/docs/teams","hooks_url":"https://api.github.com/repos/prometheus/docs/hooks","issue_events_url":"https://api.github.com/repos/prometheus/docs/issues/events{/number}","events_url":"https://api.github.com/repos/prometheus/docs/events","assignees_url":"https://api.github.com/repos/prometheus/docs/assignees{/user}","branches_url":"https://api.github.com/repos/prometheus/docs/branches{/branch}","tags_url":"https://api.github.com/repos/prometheus/docs/tags","blobs_url":"https://api.github.com/repos/prometheus/docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/prometheus/docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/prometheus/docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/prometheus/docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/prometheus/docs/statuses/{sha}","languages_url":"https://api.github.com/repos/prometheus/docs/languages","stargazers_url":"https://api.github.com/repos/prometheus/docs/stargazers","contributors_url":"https://api.github.com/repos/prometheus/docs/contributors","subscribers_url":"https://api.github.com/repos/prometheus/docs/subscribers","subscription_url":"https://api.github.com/repos/prometheus/docs/subscription","commits_url":"https://api.github.com/repos/prometheus/docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/prometheus/docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/prometheus/docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/prometheus/docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/prometheus/docs/contents/{+path}","compare_url":"https://api.github.com/repos/prometheus/docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/prometheus/docs/merges","archive_url":"https://api.github.com/repos/prometheus/docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/prometheus/docs/downloads","issues_url":"https://api.github.com/repos/prometheus/docs/issues{/number}","pulls_url":"https://api.github.com/repos/prometheus/docs/pulls{/number}","milestones_url":"https://api.github.com/repos/prometheus/docs/milestones{/number}","notifications_url":"https://api.github.com/repos/prometheus/docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/prometheus/docs/labels{/name}","releases_url":"https://api.github.com/repos/prometheus/docs/releases{/id}","created_at":"2014-12-19T15:06:35Z","updated_at":"2015-05-08T14:09:16Z","pushed_at":"2015-06-23T16:50:44Z","git_url":"git://github.com/prometheus/docs.git","ssh_url":"git@github.com:prometheus/docs.git","clone_url":"https://github.com/prometheus/docs.git","svn_url":"https://github.com/prometheus/docs","homepage":"http://prometheus.io","size":3174,"stargazers_count":6,"watchers_count":6,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":12,"mirror_url":null,"open_issues_count":2,"forks":12,"open_issues":2,"watchers":6,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/prometheus/docs/pulls/123"},"html":{"href":"https://github.com/prometheus/docs/pull/123"},"issue":{"href":"https://api.github.com/repos/prometheus/docs/issues/123"},"comments":{"href":"https://api.github.com/repos/prometheus/docs/issues/123/comments"},"review_comments":{"href":"https://api.github.com/repos/prometheus/docs/pulls/123/comments"},"review_comment":{"href":"https://api.github.com/repos/prometheus/docs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/prometheus/docs/pulls/123/commits"},"statuses":{"href":"https://api.github.com/repos/prometheus/docs/statuses/e760566ec9572f5905ce4815f6437ad0a557f053"}}}}