{"action":"created","comment":{"url":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/comments/29439203","id":29439203,"diff_hunk":"@@ -0,0 +1,537 @@\n+Quantile 1 � What is it?\n+========================\n+This series of blog posts started life as a series of walk-throughs and brainstorming sessions at a team offsite.  The\n+series will look at adding a new activity to the system.  The idea is to give an idea of the work\n+involved, to highlight the different areas that need changing, and hopefully encourage\n+others to add their own activities.  In parallel with the description in this blog there is a series of commits\n+to the github repository that correspond to the different stages in adding the activity.  Once the blog is completed,\n+the text will also be checked into the source control tree for future reference.\n+\n+The new activity is going to be a QUANTILE activity, which can be used to find the records that split a dataset into\n+equal sized blocks.  Two common uses are to find the median of a set of data (split into 2) or percentiles (split into\n+100).  It can also be used to split a dataset for distribution across the nodes in a system.  One hope is that the\n+classes used to implement quantile in Thor can also be used to improve the performance of the global sort\n+operation.\n+\n+It may seem fatuous, but the first task in adding any activity to the system is to work out what that activity is going \n+to do!  You can approach this in an iterative manner - starting with a minimal set of functionality and adding options \n+as you think of them � or start with a more complete initial design.  We have used both approaches in the past to add \n+capabilities to the HPCC system, but on this occasion we will be starting from a more complete design that came out of \n+our initial design discussion:\n+\n+\"What are the inputs, options and capabilities that might be useful in a QUANTILE activity?\"\n+\n+The following items came out of the discussion:\n+\n+* Which dataset is being processed?\n+\n+  This is always required and should be the first argument to the activity.\n+\n+* How many parts to split the dataset into?\n+\n+  This is always required, so it should be the next argument to the activity.\n+\n+* Which fields are being used to order (and split) the dataset?\n+\n+  Again this is always required, so the list of fields should follow the number of partitions.\n+\n+* Which fields are returned?\n+\n+  Normally the input row, but often it would be useful for the output to include details of which quantile a row\n+  corresponds to.  To allow this an optional transform should be that could be passed the input row as LEFT and\n+  the quantile number as COUNTER.\n+\n+* How about first and last rows in the dataset?\n+\n+  Sometimes it is also useful to know the first and last rows.  Add flags to allow them to be optionally returned.\n+\n+* How do you cope with too few input rows (including an empty input)?\n+\n+  After some discussion we decided that QUANTILE should always return the number of parts requested.  If there were fewer\n+  items in the input they would be duplicated as appropriate.  We should provide a DEDUP flag for the situations when that\n+  is not desired.\n+  If there is an empty dataset as input then the default (blank) row will be created.\n+\n+* Should all rows have the same weighting?\n+\n+  Generally you want the same weighting for each row.  However, if you are using QUANTILE to split your dataset, and the cost\n+  of the next operation depends on some feature of the row (e.g., the frequency of the firstname) then you may want to\n+  weight the rows differently.\n+\n+* What if we are only interested in the 5th and 95th centiles?\n+\n+  We could optionally allow a set of values to be selected from the results.\n+\n+There were also some implementation details that came out of the discussions:","path":"ecl/eclcc/NEWACTIVITY.rst","position":65,"original_position":65,"commit_id":"a2ff44ba3e133108977a06c4b9ef73c1d35cb94f","original_commit_id":"a2ff44ba3e133108977a06c4b9ef73c1d35cb94f","user":{"login":"jamienoss","id":5013975,"avatar_url":"https://avatars.githubusercontent.com/u/5013975?v=3","gravatar_id":"","url":"https://api.github.com/users/jamienoss","html_url":"https://github.com/jamienoss","followers_url":"https://api.github.com/users/jamienoss/followers","following_url":"https://api.github.com/users/jamienoss/following{/other_user}","gists_url":"https://api.github.com/users/jamienoss/gists{/gist_id}","starred_url":"https://api.github.com/users/jamienoss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamienoss/subscriptions","organizations_url":"https://api.github.com/users/jamienoss/orgs","repos_url":"https://api.github.com/users/jamienoss/repos","events_url":"https://api.github.com/users/jamienoss/events{/privacy}","received_events_url":"https://api.github.com/users/jamienoss/received_events","type":"User","site_admin":false},"body":"details that **_were concluded from_** the discussion","created_at":"2015-04-30T15:23:43Z","updated_at":"2015-04-30T15:23:43Z","html_url":"https://github.com/ghalliday/HPCC-Platform/pull/12#discussion_r29439203","pull_request_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/12","_links":{"self":{"href":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/comments/29439203"},"html":{"href":"https://github.com/ghalliday/HPCC-Platform/pull/12#discussion_r29439203"},"pull_request":{"href":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/12"}}},"pull_request":{"url":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/12","id":34053019,"html_url":"https://github.com/ghalliday/HPCC-Platform/pull/12","diff_url":"https://github.com/ghalliday/HPCC-Platform/pull/12.diff","patch_url":"https://github.com/ghalliday/HPCC-Platform/pull/12.patch","issue_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues/12","number":12,"state":"open","locked":false,"title":"HPCC-12267 Documentation on adding a new activity to the system","user":{"login":"ghalliday","id":902490,"avatar_url":"https://avatars.githubusercontent.com/u/902490?v=3","gravatar_id":"","url":"https://api.github.com/users/ghalliday","html_url":"https://github.com/ghalliday","followers_url":"https://api.github.com/users/ghalliday/followers","following_url":"https://api.github.com/users/ghalliday/following{/other_user}","gists_url":"https://api.github.com/users/ghalliday/gists{/gist_id}","starred_url":"https://api.github.com/users/ghalliday/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ghalliday/subscriptions","organizations_url":"https://api.github.com/users/ghalliday/orgs","repos_url":"https://api.github.com/users/ghalliday/repos","events_url":"https://api.github.com/users/ghalliday/events{/privacy}","received_events_url":"https://api.github.com/users/ghalliday/received_events","type":"User","site_admin":false},"body":"Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>","created_at":"2015-04-24T14:26:24Z","updated_at":"2015-04-30T15:23:43Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6b0019365489698780234680f9cdfea27d394820","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/12/commits","review_comments_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/12/comments","review_comment_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues/12/comments","statuses_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/statuses/a2ff44ba3e133108977a06c4b9ef73c1d35cb94f","head":{"label":"ghalliday:quantdoc","ref":"quantdoc","sha":"a2ff44ba3e133108977a06c4b9ef73c1d35cb94f","user":{"login":"ghalliday","id":902490,"avatar_url":"https://avatars.githubusercontent.com/u/902490?v=3","gravatar_id":"","url":"https://api.github.com/users/ghalliday","html_url":"https://github.com/ghalliday","followers_url":"https://api.github.com/users/ghalliday/followers","following_url":"https://api.github.com/users/ghalliday/following{/other_user}","gists_url":"https://api.github.com/users/ghalliday/gists{/gist_id}","starred_url":"https://api.github.com/users/ghalliday/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ghalliday/subscriptions","organizations_url":"https://api.github.com/users/ghalliday/orgs","repos_url":"https://api.github.com/users/ghalliday/repos","events_url":"https://api.github.com/users/ghalliday/events{/privacy}","received_events_url":"https://api.github.com/users/ghalliday/received_events","type":"User","site_admin":false},"repo":{"id":2034867,"name":"HPCC-Platform","full_name":"ghalliday/HPCC-Platform","owner":{"login":"ghalliday","id":902490,"avatar_url":"https://avatars.githubusercontent.com/u/902490?v=3","gravatar_id":"","url":"https://api.github.com/users/ghalliday","html_url":"https://github.com/ghalliday","followers_url":"https://api.github.com/users/ghalliday/followers","following_url":"https://api.github.com/users/ghalliday/following{/other_user}","gists_url":"https://api.github.com/users/ghalliday/gists{/gist_id}","starred_url":"https://api.github.com/users/ghalliday/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ghalliday/subscriptions","organizations_url":"https://api.github.com/users/ghalliday/orgs","repos_url":"https://api.github.com/users/ghalliday/repos","events_url":"https://api.github.com/users/ghalliday/events{/privacy}","received_events_url":"https://api.github.com/users/ghalliday/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ghalliday/HPCC-Platform","description":"HPCC Systems big data solution, community edition","fork":true,"url":"https://api.github.com/repos/ghalliday/HPCC-Platform","forks_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/forks","keys_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/teams","hooks_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/hooks","issue_events_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues/events{/number}","events_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/events","assignees_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/assignees{/user}","branches_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/branches{/branch}","tags_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/tags","blobs_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/refs{/sha}","trees_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/statuses/{sha}","languages_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/languages","stargazers_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/stargazers","contributors_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/contributors","subscribers_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/subscribers","subscription_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/subscription","commits_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/commits{/sha}","git_commits_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/commits{/sha}","comments_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/comments{/number}","issue_comment_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues/comments{/number}","contents_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/contents/{+path}","compare_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/merges","archive_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/downloads","issues_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues{/number}","pulls_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls{/number}","milestones_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/milestones{/number}","notifications_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/labels{/name}","releases_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/releases{/id}","created_at":"2011-07-12T08:05:04Z","updated_at":"2015-04-09T16:09:15Z","pushed_at":"2015-04-30T12:38:46Z","git_url":"git://github.com/ghalliday/HPCC-Platform.git","ssh_url":"git@github.com:ghalliday/HPCC-Platform.git","clone_url":"https://github.com/ghalliday/HPCC-Platform.git","svn_url":"https://github.com/ghalliday/HPCC-Platform","homepage":"www.hpccsystems.com","size":150702,"stargazers_count":2,"watchers_count":2,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":2,"default_branch":"master"}},"base":{"label":"ghalliday:master","ref":"master","sha":"f38398c7034d51d8ad7c94489a3dde42cd6df86a","user":{"login":"ghalliday","id":902490,"avatar_url":"https://avatars.githubusercontent.com/u/902490?v=3","gravatar_id":"","url":"https://api.github.com/users/ghalliday","html_url":"https://github.com/ghalliday","followers_url":"https://api.github.com/users/ghalliday/followers","following_url":"https://api.github.com/users/ghalliday/following{/other_user}","gists_url":"https://api.github.com/users/ghalliday/gists{/gist_id}","starred_url":"https://api.github.com/users/ghalliday/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ghalliday/subscriptions","organizations_url":"https://api.github.com/users/ghalliday/orgs","repos_url":"https://api.github.com/users/ghalliday/repos","events_url":"https://api.github.com/users/ghalliday/events{/privacy}","received_events_url":"https://api.github.com/users/ghalliday/received_events","type":"User","site_admin":false},"repo":{"id":2034867,"name":"HPCC-Platform","full_name":"ghalliday/HPCC-Platform","owner":{"login":"ghalliday","id":902490,"avatar_url":"https://avatars.githubusercontent.com/u/902490?v=3","gravatar_id":"","url":"https://api.github.com/users/ghalliday","html_url":"https://github.com/ghalliday","followers_url":"https://api.github.com/users/ghalliday/followers","following_url":"https://api.github.com/users/ghalliday/following{/other_user}","gists_url":"https://api.github.com/users/ghalliday/gists{/gist_id}","starred_url":"https://api.github.com/users/ghalliday/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ghalliday/subscriptions","organizations_url":"https://api.github.com/users/ghalliday/orgs","repos_url":"https://api.github.com/users/ghalliday/repos","events_url":"https://api.github.com/users/ghalliday/events{/privacy}","received_events_url":"https://api.github.com/users/ghalliday/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ghalliday/HPCC-Platform","description":"HPCC Systems big data solution, community edition","fork":true,"url":"https://api.github.com/repos/ghalliday/HPCC-Platform","forks_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/forks","keys_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/teams","hooks_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/hooks","issue_events_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues/events{/number}","events_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/events","assignees_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/assignees{/user}","branches_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/branches{/branch}","tags_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/tags","blobs_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/refs{/sha}","trees_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/statuses/{sha}","languages_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/languages","stargazers_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/stargazers","contributors_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/contributors","subscribers_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/subscribers","subscription_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/subscription","commits_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/commits{/sha}","git_commits_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/git/commits{/sha}","comments_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/comments{/number}","issue_comment_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues/comments{/number}","contents_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/contents/{+path}","compare_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/merges","archive_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/downloads","issues_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues{/number}","pulls_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls{/number}","milestones_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/milestones{/number}","notifications_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/labels{/name}","releases_url":"https://api.github.com/repos/ghalliday/HPCC-Platform/releases{/id}","created_at":"2011-07-12T08:05:04Z","updated_at":"2015-04-09T16:09:15Z","pushed_at":"2015-04-30T12:38:46Z","git_url":"git://github.com/ghalliday/HPCC-Platform.git","ssh_url":"git@github.com:ghalliday/HPCC-Platform.git","clone_url":"https://github.com/ghalliday/HPCC-Platform.git","svn_url":"https://github.com/ghalliday/HPCC-Platform","homepage":"www.hpccsystems.com","size":150702,"stargazers_count":2,"watchers_count":2,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/12"},"html":{"href":"https://github.com/ghalliday/HPCC-Platform/pull/12"},"issue":{"href":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues/12"},"comments":{"href":"https://api.github.com/repos/ghalliday/HPCC-Platform/issues/12/comments"},"review_comments":{"href":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/12/comments"},"review_comment":{"href":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ghalliday/HPCC-Platform/pulls/12/commits"},"statuses":{"href":"https://api.github.com/repos/ghalliday/HPCC-Platform/statuses/a2ff44ba3e133108977a06c4b9ef73c1d35cb94f"}}}}