{"action":"created","comment":{"url":"https://api.github.com/repos/apache/flink/pulls/comments/37144171","id":37144171,"diff_hunk":"@@ -0,0 +1,100 @@\n+---\n+mathjax: include\n+htmlTitle: FlinkML - Statistics\n+title: <a href=\"../ml\">FlinkML</a> - Statistics\n+---\n+<!--\n+Licensed to the Apache Software Foundation (ASF) under one\n+or more contributor license agreements.  See the NOTICE file\n+distributed with this work for additional information\n+regarding copyright ownership.  The ASF licenses this file\n+to you under the Apache License, Version 2.0 (the\n+\"License\"); you may not use this file except in compliance\n+with the License.  You may obtain a copy of the License at\n+\n+  http://www.apache.org/licenses/LICENSE-2.0\n+\n+Unless required by applicable law or agreed to in writing,\n+software distributed under the License is distributed on an\n+\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n+KIND, either express or implied.  See the License for the\n+specific language governing permissions and limitations\n+under the License.\n+-->\n+\n+* This will be replaced by the TOC\n+{:toc}\n+\n+## Description\n+\n+ The statistics utility provides features such as building histograms over data, determining\n+ mean, variance, gini impurity, entropy etc. of data.\n+\n+## Methods\n+\n+ The Statistics utility provides two major functions: `createHistogram` and `dataStats`.\n+\n+### Creating a histogram\n+\n+ There are two types of histograms:\n+   1. <strong>Continuous Histograms</strong>: These histograms are formed on a data set `X:\n+   DataSet[Double]`\n+   when the values in `X` are from a continuous range. These histograms support\n+   `quantile` and `sum`  operations. Here `quantile(q)` refers to a value $x_q$ such that $|x: x\n+   \\leq x_q| = q * |X|$. Further, `sum(s)` refers to the number of elements $x \\leq s$, which can\n+    be construed as a cumulative probability value at $s$[Of course, <i>scaled</i> probability].\n+   <br>\n+   2. A continuous histogram can be formed by calling `X.createHistogram(b)` where `b` is the\n+    number of bins.\n+    <strong>Categorical Histograms</strong>: These histograms are formed on a data set `X:DataSet[Double]` ","path":"docs/libs/ml/statistics.md","position":49,"original_position":49,"commit_id":"ee57f3b48ea3ce8b647f0bf76026050f3967a82a","original_commit_id":"ee57f3b48ea3ce8b647f0bf76026050f3967a82a","user":{"login":"chiwanpark","id":1941681,"avatar_url":"https://avatars.githubusercontent.com/u/1941681?v=3","gravatar_id":"","url":"https://api.github.com/users/chiwanpark","html_url":"https://github.com/chiwanpark","followers_url":"https://api.github.com/users/chiwanpark/followers","following_url":"https://api.github.com/users/chiwanpark/following{/other_user}","gists_url":"https://api.github.com/users/chiwanpark/gists{/gist_id}","starred_url":"https://api.github.com/users/chiwanpark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chiwanpark/subscriptions","organizations_url":"https://api.github.com/users/chiwanpark/orgs","repos_url":"https://api.github.com/users/chiwanpark/repos","events_url":"https://api.github.com/users/chiwanpark/events{/privacy}","received_events_url":"https://api.github.com/users/chiwanpark/received_events","type":"User","site_admin":false},"body":"`<strong>` tag","created_at":"2015-08-16T09:43:25Z","updated_at":"2015-08-16T09:43:25Z","html_url":"https://github.com/apache/flink/pull/861#discussion_r37144171","pull_request_url":"https://api.github.com/repos/apache/flink/pulls/861","_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/comments/37144171"},"html":{"href":"https://github.com/apache/flink/pull/861#discussion_r37144171"},"pull_request":{"href":"https://api.github.com/repos/apache/flink/pulls/861"}}},"pull_request":{"url":"https://api.github.com/repos/apache/flink/pulls/861","id":38313488,"html_url":"https://github.com/apache/flink/pull/861","diff_url":"https://github.com/apache/flink/pull/861.diff","patch_url":"https://github.com/apache/flink/pull/861.patch","issue_url":"https://api.github.com/repos/apache/flink/issues/861","number":861,"state":"open","locked":false,"title":"[Flink-2030][ml]Data Set Statistics and Histograms","user":{"login":"sachingoel0101","id":8874261,"avatar_url":"https://avatars.githubusercontent.com/u/8874261?v=3","gravatar_id":"","url":"https://api.github.com/users/sachingoel0101","html_url":"https://github.com/sachingoel0101","followers_url":"https://api.github.com/users/sachingoel0101/followers","following_url":"https://api.github.com/users/sachingoel0101/following{/other_user}","gists_url":"https://api.github.com/users/sachingoel0101/gists{/gist_id}","starred_url":"https://api.github.com/users/sachingoel0101/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sachingoel0101/subscriptions","organizations_url":"https://api.github.com/users/sachingoel0101/orgs","repos_url":"https://api.github.com/users/sachingoel0101/repos","events_url":"https://api.github.com/users/sachingoel0101/events{/privacy}","received_events_url":"https://api.github.com/users/sachingoel0101/received_events","type":"User","site_admin":false},"body":"This implements the Online Histograms for both categorical and continuous data. For continuous data, we emulate a continuous probability distribution which supports finding cumulative sum upto a particular value, and finding value upto a specific cumulative probability [Quantiles]. \r\nFor categorical fields, we emulate a probability mass function which supports finding the probability associated with every class.\r\nThe continuous histogram follows this paper: http://www.jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf\r\n\r\nNote: This is a sub-task of https://issues.apache.org/jira/browse/FLINK-1727 which already has a PR pending review at https://github.com/apache/flink/pull/710.\r\n\r\nEdit: This adds methods to evaluate statistics on data sets of vectors, like column wise statistics. These include minimum, maximum, mean, variance, entropy, gini impurity, etc. [FLINK-2379]","created_at":"2015-06-23T08:47:48Z","updated_at":"2015-08-16T09:43:26Z","closed_at":null,"merged_at":null,"merge_commit_sha":"434da88be128cda92147afeaa7bade1a7e0c70d8","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/flink/pulls/861/commits","review_comments_url":"https://api.github.com/repos/apache/flink/pulls/861/comments","review_comment_url":"https://api.github.com/repos/apache/flink/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/flink/issues/861/comments","statuses_url":"https://api.github.com/repos/apache/flink/statuses/ee57f3b48ea3ce8b647f0bf76026050f3967a82a","head":{"label":"sachingoel0101:online_histogram","ref":"online_histogram","sha":"ee57f3b48ea3ce8b647f0bf76026050f3967a82a","user":{"login":"sachingoel0101","id":8874261,"avatar_url":"https://avatars.githubusercontent.com/u/8874261?v=3","gravatar_id":"","url":"https://api.github.com/users/sachingoel0101","html_url":"https://github.com/sachingoel0101","followers_url":"https://api.github.com/users/sachingoel0101/followers","following_url":"https://api.github.com/users/sachingoel0101/following{/other_user}","gists_url":"https://api.github.com/users/sachingoel0101/gists{/gist_id}","starred_url":"https://api.github.com/users/sachingoel0101/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sachingoel0101/subscriptions","organizations_url":"https://api.github.com/users/sachingoel0101/orgs","repos_url":"https://api.github.com/users/sachingoel0101/repos","events_url":"https://api.github.com/users/sachingoel0101/events{/privacy}","received_events_url":"https://api.github.com/users/sachingoel0101/received_events","type":"User","site_admin":false},"repo":{"id":35545853,"name":"flink","full_name":"sachingoel0101/flink","owner":{"login":"sachingoel0101","id":8874261,"avatar_url":"https://avatars.githubusercontent.com/u/8874261?v=3","gravatar_id":"","url":"https://api.github.com/users/sachingoel0101","html_url":"https://github.com/sachingoel0101","followers_url":"https://api.github.com/users/sachingoel0101/followers","following_url":"https://api.github.com/users/sachingoel0101/following{/other_user}","gists_url":"https://api.github.com/users/sachingoel0101/gists{/gist_id}","starred_url":"https://api.github.com/users/sachingoel0101/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sachingoel0101/subscriptions","organizations_url":"https://api.github.com/users/sachingoel0101/orgs","repos_url":"https://api.github.com/users/sachingoel0101/repos","events_url":"https://api.github.com/users/sachingoel0101/events{/privacy}","received_events_url":"https://api.github.com/users/sachingoel0101/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/sachingoel0101/flink","description":"Mirror of Apache Flink","fork":true,"url":"https://api.github.com/repos/sachingoel0101/flink","forks_url":"https://api.github.com/repos/sachingoel0101/flink/forks","keys_url":"https://api.github.com/repos/sachingoel0101/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/sachingoel0101/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/sachingoel0101/flink/teams","hooks_url":"https://api.github.com/repos/sachingoel0101/flink/hooks","issue_events_url":"https://api.github.com/repos/sachingoel0101/flink/issues/events{/number}","events_url":"https://api.github.com/repos/sachingoel0101/flink/events","assignees_url":"https://api.github.com/repos/sachingoel0101/flink/assignees{/user}","branches_url":"https://api.github.com/repos/sachingoel0101/flink/branches{/branch}","tags_url":"https://api.github.com/repos/sachingoel0101/flink/tags","blobs_url":"https://api.github.com/repos/sachingoel0101/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/sachingoel0101/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/sachingoel0101/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/sachingoel0101/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/sachingoel0101/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/sachingoel0101/flink/languages","stargazers_url":"https://api.github.com/repos/sachingoel0101/flink/stargazers","contributors_url":"https://api.github.com/repos/sachingoel0101/flink/contributors","subscribers_url":"https://api.github.com/repos/sachingoel0101/flink/subscribers","subscription_url":"https://api.github.com/repos/sachingoel0101/flink/subscription","commits_url":"https://api.github.com/repos/sachingoel0101/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/sachingoel0101/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/sachingoel0101/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/sachingoel0101/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/sachingoel0101/flink/contents/{+path}","compare_url":"https://api.github.com/repos/sachingoel0101/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/sachingoel0101/flink/merges","archive_url":"https://api.github.com/repos/sachingoel0101/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/sachingoel0101/flink/downloads","issues_url":"https://api.github.com/repos/sachingoel0101/flink/issues{/number}","pulls_url":"https://api.github.com/repos/sachingoel0101/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/sachingoel0101/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/sachingoel0101/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/sachingoel0101/flink/labels{/name}","releases_url":"https://api.github.com/repos/sachingoel0101/flink/releases{/id}","created_at":"2015-05-13T11:44:31Z","updated_at":"2015-05-13T11:44:41Z","pushed_at":"2015-08-16T09:24:47Z","git_url":"git://github.com/sachingoel0101/flink.git","ssh_url":"git@github.com:sachingoel0101/flink.git","clone_url":"https://github.com/sachingoel0101/flink.git","svn_url":"https://github.com/sachingoel0101/flink","homepage":null,"size":56482,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"c2b1eb7961ed16b8985d7f8c02e87c0964b818e8","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":20587599,"name":"flink","full_name":"apache/flink","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/flink","description":"Mirror of Apache Flink","fork":false,"url":"https://api.github.com/repos/apache/flink","forks_url":"https://api.github.com/repos/apache/flink/forks","keys_url":"https://api.github.com/repos/apache/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/flink/teams","hooks_url":"https://api.github.com/repos/apache/flink/hooks","issue_events_url":"https://api.github.com/repos/apache/flink/issues/events{/number}","events_url":"https://api.github.com/repos/apache/flink/events","assignees_url":"https://api.github.com/repos/apache/flink/assignees{/user}","branches_url":"https://api.github.com/repos/apache/flink/branches{/branch}","tags_url":"https://api.github.com/repos/apache/flink/tags","blobs_url":"https://api.github.com/repos/apache/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/flink/languages","stargazers_url":"https://api.github.com/repos/apache/flink/stargazers","contributors_url":"https://api.github.com/repos/apache/flink/contributors","subscribers_url":"https://api.github.com/repos/apache/flink/subscribers","subscription_url":"https://api.github.com/repos/apache/flink/subscription","commits_url":"https://api.github.com/repos/apache/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/flink/contents/{+path}","compare_url":"https://api.github.com/repos/apache/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/flink/merges","archive_url":"https://api.github.com/repos/apache/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/flink/downloads","issues_url":"https://api.github.com/repos/apache/flink/issues{/number}","pulls_url":"https://api.github.com/repos/apache/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/flink/labels{/name}","releases_url":"https://api.github.com/repos/apache/flink/releases{/id}","created_at":"2014-06-07T07:00:10Z","updated_at":"2015-08-14T08:19:41Z","pushed_at":"2015-08-16T09:24:47Z","git_url":"git://github.com/apache/flink.git","ssh_url":"git@github.com:apache/flink.git","clone_url":"https://github.com/apache/flink.git","svn_url":"https://github.com/apache/flink","homepage":null,"size":2606770,"stargazers_count":562,"watchers_count":562,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":369,"mirror_url":"git://git.apache.org/incubator-flink.git","open_issues_count":78,"forks":369,"open_issues":78,"watchers":562,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/861"},"html":{"href":"https://github.com/apache/flink/pull/861"},"issue":{"href":"https://api.github.com/repos/apache/flink/issues/861"},"comments":{"href":"https://api.github.com/repos/apache/flink/issues/861/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/flink/pulls/861/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/flink/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/flink/pulls/861/commits"},"statuses":{"href":"https://api.github.com/repos/apache/flink/statuses/ee57f3b48ea3ce8b647f0bf76026050f3967a82a"}}}}