{"action":"created","comment":{"url":"https://api.github.com/repos/datarank/dev-blog/pulls/comments/33518465","id":33518465,"diff_hunk":"@@ -0,0 +1,125 @@\n+---\n+layout: post\n+title: \"Analysis of Hotspots in Clusters of Log-Normally Distributed Data\"\n+author: Andrew White\n+date: 2015-06-16T20:11:08-05:00\n+---\n+\n+This post will be exploring the statistical nature of log normal distributed data with an emphasis on how it affects the\n+ design of Elasticsearch clusters. However, the data and conclusions should be relevant beyond the scope of Elasticsearch.\n+\n+<!-- loading MathJax here unless there is a better way elsewhere -->\n+<script type=\"text/javascript\"\n+  src=\"https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML\">\n+</script>\n+\n+First, let's define some terms. At Datarank, we track \"topics\". A topic is nothing more than a collections of comments\n+that match some criteria. These criteria can be rather broad to very specific. A shard is a slice of an Elasitcsearch\n+index that holds multiple topics. An index is made up of multiple shards. We will use certain statistical shorthands as\n+well such as \"PDF\" to mean a probability distribution function, \"stddev/σ\" for standard deviation. Lastly, all graphs\n+based on empirical data were first mapped to a smooth kernel distribution with 1/2 σ bandwidth.\n+\n+## Topic Sizes\n+\n+First, lets look at topic sizes. Topics ranged in size from very tiny to very large with most having between 10k and 100k\n+comments. This looked like a classic log-normal distribution and sure enough it was with a mean of 9.022 and a σ of 2.832.\n+\n+<!--\n+  -- switching to raw HTML because...\n+  -- Liquid Exception: Unknown tag 'img'\n+  -->\n+<img src=\"/images/posts/2015-06-16/pdf-topic-size.png\">\n+\n+Since we will be referencing this distribution later and to save a trip to a stats textbook or Wikipedia, the PDF for\n+topic sizes can be modeled using the following equation.\n+\n+ $$P(T)=\\frac{1}{x\\sigma\\sqrt{2\\pi}}e^{-\\frac{(\\ln x - \\mu)^2}{2 \\sigma^2}};\\;\\mu=9.022\\,\\mathrm{and}\\,\\sigma=2.932$$\n+\n+As an interesting aside, this is the same equation used to model the spread of highly communicable epidemics. Since\n+topics track abstract concepts, this may provide some good evidence for modeling highly shared content, such as ads and\n+memes, as viruses.\n+\n+## Shard Sizes\n+\n+For performance reasons, we like to map a topic to a minimum number of shards, ideally one. To that end we use\n+Elasticsearch's routing feature and use the topic's numerical id as the routing key. It's important to note that routing\n+is static so that document placement is predictable and stable as more data is added to the index.\n+\n+Since a shard is simply a collections of topics, it's size PDF can be expressed as follows.\n+\n+ $$P(S) = T + T + T + ... + T$$","path":"_posts/2015-06-16-analysis-of-hotspots-in-clusters-of-log-normally-distributed-data.markdown","position":50,"original_position":50,"commit_id":"40896c2e2b76e7d78e0bb149a3c1a90676e7c861","original_commit_id":"40896c2e2b76e7d78e0bb149a3c1a90676e7c861","user":{"login":"aewhite","id":4342706,"avatar_url":"https://avatars.githubusercontent.com/u/4342706?v=3","gravatar_id":"","url":"https://api.github.com/users/aewhite","html_url":"https://github.com/aewhite","followers_url":"https://api.github.com/users/aewhite/followers","following_url":"https://api.github.com/users/aewhite/following{/other_user}","gists_url":"https://api.github.com/users/aewhite/gists{/gist_id}","starred_url":"https://api.github.com/users/aewhite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aewhite/subscriptions","organizations_url":"https://api.github.com/users/aewhite/orgs","repos_url":"https://api.github.com/users/aewhite/repos","events_url":"https://api.github.com/users/aewhite/events{/privacy}","received_events_url":"https://api.github.com/users/aewhite/received_events","type":"User","site_admin":false},"body":"I've seen it written a couple of ways and probably S = T + T + T is better (see http://leo.ugr.es/pgm2012/submissions/pgm2012_submission_6.pdf)","created_at":"2015-06-29T21:56:51Z","updated_at":"2015-06-29T21:56:51Z","html_url":"https://github.com/datarank/dev-blog/pull/3#discussion_r33518465","pull_request_url":"https://api.github.com/repos/datarank/dev-blog/pulls/3","_links":{"self":{"href":"https://api.github.com/repos/datarank/dev-blog/pulls/comments/33518465"},"html":{"href":"https://github.com/datarank/dev-blog/pull/3#discussion_r33518465"},"pull_request":{"href":"https://api.github.com/repos/datarank/dev-blog/pulls/3"}}},"pull_request":{"url":"https://api.github.com/repos/datarank/dev-blog/pulls/3","id":38817914,"html_url":"https://github.com/datarank/dev-blog/pull/3","diff_url":"https://github.com/datarank/dev-blog/pull/3.diff","patch_url":"https://github.com/datarank/dev-blog/pull/3.patch","issue_url":"https://api.github.com/repos/datarank/dev-blog/issues/3","number":3,"state":"open","locked":false,"title":"2015-03-15 ES Hotspots","user":{"login":"aewhite","id":4342706,"avatar_url":"https://avatars.githubusercontent.com/u/4342706?v=3","gravatar_id":"","url":"https://api.github.com/users/aewhite","html_url":"https://github.com/aewhite","followers_url":"https://api.github.com/users/aewhite/followers","following_url":"https://api.github.com/users/aewhite/following{/other_user}","gists_url":"https://api.github.com/users/aewhite/gists{/gist_id}","starred_url":"https://api.github.com/users/aewhite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aewhite/subscriptions","organizations_url":"https://api.github.com/users/aewhite/orgs","repos_url":"https://api.github.com/users/aewhite/repos","events_url":"https://api.github.com/users/aewhite/events{/privacy}","received_events_url":"https://api.github.com/users/aewhite/received_events","type":"User","site_admin":false},"body":"@kennycason @addamh review please \r\n\r\nYou will probably want to review this in an HTML preview. ","created_at":"2015-06-29T21:09:11Z","updated_at":"2015-06-29T21:56:51Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d47c2340a893d115505f4bbbe2c188ecd4f23dd3","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/datarank/dev-blog/pulls/3/commits","review_comments_url":"https://api.github.com/repos/datarank/dev-blog/pulls/3/comments","review_comment_url":"https://api.github.com/repos/datarank/dev-blog/pulls/comments{/number}","comments_url":"https://api.github.com/repos/datarank/dev-blog/issues/3/comments","statuses_url":"https://api.github.com/repos/datarank/dev-blog/statuses/40896c2e2b76e7d78e0bb149a3c1a90676e7c861","head":{"label":"datarank:high-varience-hotspots","ref":"high-varience-hotspots","sha":"40896c2e2b76e7d78e0bb149a3c1a90676e7c861","user":{"login":"datarank","id":7254554,"avatar_url":"https://avatars.githubusercontent.com/u/7254554?v=3","gravatar_id":"","url":"https://api.github.com/users/datarank","html_url":"https://github.com/datarank","followers_url":"https://api.github.com/users/datarank/followers","following_url":"https://api.github.com/users/datarank/following{/other_user}","gists_url":"https://api.github.com/users/datarank/gists{/gist_id}","starred_url":"https://api.github.com/users/datarank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datarank/subscriptions","organizations_url":"https://api.github.com/users/datarank/orgs","repos_url":"https://api.github.com/users/datarank/repos","events_url":"https://api.github.com/users/datarank/events{/privacy}","received_events_url":"https://api.github.com/users/datarank/received_events","type":"Organization","site_admin":false},"repo":{"id":31028408,"name":"dev-blog","full_name":"datarank/dev-blog","owner":{"login":"datarank","id":7254554,"avatar_url":"https://avatars.githubusercontent.com/u/7254554?v=3","gravatar_id":"","url":"https://api.github.com/users/datarank","html_url":"https://github.com/datarank","followers_url":"https://api.github.com/users/datarank/followers","following_url":"https://api.github.com/users/datarank/following{/other_user}","gists_url":"https://api.github.com/users/datarank/gists{/gist_id}","starred_url":"https://api.github.com/users/datarank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datarank/subscriptions","organizations_url":"https://api.github.com/users/datarank/orgs","repos_url":"https://api.github.com/users/datarank/repos","events_url":"https://api.github.com/users/datarank/events{/privacy}","received_events_url":"https://api.github.com/users/datarank/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/datarank/dev-blog","description":null,"fork":false,"url":"https://api.github.com/repos/datarank/dev-blog","forks_url":"https://api.github.com/repos/datarank/dev-blog/forks","keys_url":"https://api.github.com/repos/datarank/dev-blog/keys{/key_id}","collaborators_url":"https://api.github.com/repos/datarank/dev-blog/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/datarank/dev-blog/teams","hooks_url":"https://api.github.com/repos/datarank/dev-blog/hooks","issue_events_url":"https://api.github.com/repos/datarank/dev-blog/issues/events{/number}","events_url":"https://api.github.com/repos/datarank/dev-blog/events","assignees_url":"https://api.github.com/repos/datarank/dev-blog/assignees{/user}","branches_url":"https://api.github.com/repos/datarank/dev-blog/branches{/branch}","tags_url":"https://api.github.com/repos/datarank/dev-blog/tags","blobs_url":"https://api.github.com/repos/datarank/dev-blog/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/datarank/dev-blog/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/datarank/dev-blog/git/refs{/sha}","trees_url":"https://api.github.com/repos/datarank/dev-blog/git/trees{/sha}","statuses_url":"https://api.github.com/repos/datarank/dev-blog/statuses/{sha}","languages_url":"https://api.github.com/repos/datarank/dev-blog/languages","stargazers_url":"https://api.github.com/repos/datarank/dev-blog/stargazers","contributors_url":"https://api.github.com/repos/datarank/dev-blog/contributors","subscribers_url":"https://api.github.com/repos/datarank/dev-blog/subscribers","subscription_url":"https://api.github.com/repos/datarank/dev-blog/subscription","commits_url":"https://api.github.com/repos/datarank/dev-blog/commits{/sha}","git_commits_url":"https://api.github.com/repos/datarank/dev-blog/git/commits{/sha}","comments_url":"https://api.github.com/repos/datarank/dev-blog/comments{/number}","issue_comment_url":"https://api.github.com/repos/datarank/dev-blog/issues/comments{/number}","contents_url":"https://api.github.com/repos/datarank/dev-blog/contents/{+path}","compare_url":"https://api.github.com/repos/datarank/dev-blog/compare/{base}...{head}","merges_url":"https://api.github.com/repos/datarank/dev-blog/merges","archive_url":"https://api.github.com/repos/datarank/dev-blog/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/datarank/dev-blog/downloads","issues_url":"https://api.github.com/repos/datarank/dev-blog/issues{/number}","pulls_url":"https://api.github.com/repos/datarank/dev-blog/pulls{/number}","milestones_url":"https://api.github.com/repos/datarank/dev-blog/milestones{/number}","notifications_url":"https://api.github.com/repos/datarank/dev-blog/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/datarank/dev-blog/labels{/name}","releases_url":"https://api.github.com/repos/datarank/dev-blog/releases{/id}","created_at":"2015-02-19T17:54:36Z","updated_at":"2015-05-02T02:41:20Z","pushed_at":"2015-06-29T21:09:12Z","git_url":"git://github.com/datarank/dev-blog.git","ssh_url":"git@github.com:datarank/dev-blog.git","clone_url":"https://github.com/datarank/dev-blog.git","svn_url":"https://github.com/datarank/dev-blog","homepage":null,"size":1596,"stargazers_count":0,"watchers_count":0,"language":"HTML","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"datarank:master","ref":"master","sha":"589860ddf346ecbcd35f81487cf102509976fda9","user":{"login":"datarank","id":7254554,"avatar_url":"https://avatars.githubusercontent.com/u/7254554?v=3","gravatar_id":"","url":"https://api.github.com/users/datarank","html_url":"https://github.com/datarank","followers_url":"https://api.github.com/users/datarank/followers","following_url":"https://api.github.com/users/datarank/following{/other_user}","gists_url":"https://api.github.com/users/datarank/gists{/gist_id}","starred_url":"https://api.github.com/users/datarank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datarank/subscriptions","organizations_url":"https://api.github.com/users/datarank/orgs","repos_url":"https://api.github.com/users/datarank/repos","events_url":"https://api.github.com/users/datarank/events{/privacy}","received_events_url":"https://api.github.com/users/datarank/received_events","type":"Organization","site_admin":false},"repo":{"id":31028408,"name":"dev-blog","full_name":"datarank/dev-blog","owner":{"login":"datarank","id":7254554,"avatar_url":"https://avatars.githubusercontent.com/u/7254554?v=3","gravatar_id":"","url":"https://api.github.com/users/datarank","html_url":"https://github.com/datarank","followers_url":"https://api.github.com/users/datarank/followers","following_url":"https://api.github.com/users/datarank/following{/other_user}","gists_url":"https://api.github.com/users/datarank/gists{/gist_id}","starred_url":"https://api.github.com/users/datarank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datarank/subscriptions","organizations_url":"https://api.github.com/users/datarank/orgs","repos_url":"https://api.github.com/users/datarank/repos","events_url":"https://api.github.com/users/datarank/events{/privacy}","received_events_url":"https://api.github.com/users/datarank/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/datarank/dev-blog","description":null,"fork":false,"url":"https://api.github.com/repos/datarank/dev-blog","forks_url":"https://api.github.com/repos/datarank/dev-blog/forks","keys_url":"https://api.github.com/repos/datarank/dev-blog/keys{/key_id}","collaborators_url":"https://api.github.com/repos/datarank/dev-blog/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/datarank/dev-blog/teams","hooks_url":"https://api.github.com/repos/datarank/dev-blog/hooks","issue_events_url":"https://api.github.com/repos/datarank/dev-blog/issues/events{/number}","events_url":"https://api.github.com/repos/datarank/dev-blog/events","assignees_url":"https://api.github.com/repos/datarank/dev-blog/assignees{/user}","branches_url":"https://api.github.com/repos/datarank/dev-blog/branches{/branch}","tags_url":"https://api.github.com/repos/datarank/dev-blog/tags","blobs_url":"https://api.github.com/repos/datarank/dev-blog/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/datarank/dev-blog/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/datarank/dev-blog/git/refs{/sha}","trees_url":"https://api.github.com/repos/datarank/dev-blog/git/trees{/sha}","statuses_url":"https://api.github.com/repos/datarank/dev-blog/statuses/{sha}","languages_url":"https://api.github.com/repos/datarank/dev-blog/languages","stargazers_url":"https://api.github.com/repos/datarank/dev-blog/stargazers","contributors_url":"https://api.github.com/repos/datarank/dev-blog/contributors","subscribers_url":"https://api.github.com/repos/datarank/dev-blog/subscribers","subscription_url":"https://api.github.com/repos/datarank/dev-blog/subscription","commits_url":"https://api.github.com/repos/datarank/dev-blog/commits{/sha}","git_commits_url":"https://api.github.com/repos/datarank/dev-blog/git/commits{/sha}","comments_url":"https://api.github.com/repos/datarank/dev-blog/comments{/number}","issue_comment_url":"https://api.github.com/repos/datarank/dev-blog/issues/comments{/number}","contents_url":"https://api.github.com/repos/datarank/dev-blog/contents/{+path}","compare_url":"https://api.github.com/repos/datarank/dev-blog/compare/{base}...{head}","merges_url":"https://api.github.com/repos/datarank/dev-blog/merges","archive_url":"https://api.github.com/repos/datarank/dev-blog/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/datarank/dev-blog/downloads","issues_url":"https://api.github.com/repos/datarank/dev-blog/issues{/number}","pulls_url":"https://api.github.com/repos/datarank/dev-blog/pulls{/number}","milestones_url":"https://api.github.com/repos/datarank/dev-blog/milestones{/number}","notifications_url":"https://api.github.com/repos/datarank/dev-blog/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/datarank/dev-blog/labels{/name}","releases_url":"https://api.github.com/repos/datarank/dev-blog/releases{/id}","created_at":"2015-02-19T17:54:36Z","updated_at":"2015-05-02T02:41:20Z","pushed_at":"2015-06-29T21:09:12Z","git_url":"git://github.com/datarank/dev-blog.git","ssh_url":"git@github.com:datarank/dev-blog.git","clone_url":"https://github.com/datarank/dev-blog.git","svn_url":"https://github.com/datarank/dev-blog","homepage":null,"size":1596,"stargazers_count":0,"watchers_count":0,"language":"HTML","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/datarank/dev-blog/pulls/3"},"html":{"href":"https://github.com/datarank/dev-blog/pull/3"},"issue":{"href":"https://api.github.com/repos/datarank/dev-blog/issues/3"},"comments":{"href":"https://api.github.com/repos/datarank/dev-blog/issues/3/comments"},"review_comments":{"href":"https://api.github.com/repos/datarank/dev-blog/pulls/3/comments"},"review_comment":{"href":"https://api.github.com/repos/datarank/dev-blog/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/datarank/dev-blog/pulls/3/commits"},"statuses":{"href":"https://api.github.com/repos/datarank/dev-blog/statuses/40896c2e2b76e7d78e0bb149a3c1a90676e7c861"}}}}