{"action":"created","comment":{"url":"https://api.github.com/repos/kite-sdk/kite/pulls/comments/31771302","id":31771302,"diff_hunk":"@@ -191,13 +191,43 @@ public static Target asTarget(URI uri) {\n   public static <E> PCollection<E> partition(PCollection<E> collection,\n                                              View<E> view,\n                                              int numWriters) {\n+    return partition(collection, view, numWriters, 1);\n+  }\n+\n+  /**\n+   * Partitions {@code collection} to be stored efficiently in {@code View}.\n+   * <p>\n+   * This restructures the parallel collection so that all of the entities that\n+   * will be stored in a given partition will be evenly distributed across a specified\n+   * {@code numPartitionWriters}.\n+   * <p>\n+   * If the dataset is not partitioned, then this will structure all of the\n+   * entities to produce a number of files equal to {@code numWriters}.\n+   *\n+   * @param collection a collection of entities\n+   * @param view a {@link View} of a dataset to partition the collection for\n+   * @param numWriters the number of writers that should be used\n+   * @param numPartitionWriters the number of writers data for a single partition will be distributed across\n+   * @param <E> the type of entities in the collection and underlying dataset\n+   * @return an equivalent collection of entities partitioned for the view\n+   * @see #partition(PCollection, View)\n+   *\n+   * @since 1.1.0\n+   */\n+  public static <E> PCollection<E> partition(PCollection<E> collection,\n+                                             View<E> view,\n+                                             int numWriters, int numPartitionWriters) {\n+\n+    //ensure the number of writers is honored whether it is per partition or total.\n+    int writers = Math.max(numPartitionWriters, numWriters);","path":"kite-data/kite-data-crunch/src/main/java/org/kitesdk/data/crunch/CrunchDatasets.java","position":null,"original_position":32,"commit_id":"3fd7ab854fe6f7f155c6993a2a2a129f7188e0ed","original_commit_id":"8b9f5925f1a83d5439c6c33aa369165f0c67a751","user":{"login":"rdblue","id":87915,"avatar_url":"https://avatars.githubusercontent.com/u/87915?v=3","gravatar_id":"","url":"https://api.github.com/users/rdblue","html_url":"https://github.com/rdblue","followers_url":"https://api.github.com/users/rdblue/followers","following_url":"https://api.github.com/users/rdblue/following{/other_user}","gists_url":"https://api.github.com/users/rdblue/gists{/gist_id}","starred_url":"https://api.github.com/users/rdblue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rdblue/subscriptions","organizations_url":"https://api.github.com/users/rdblue/orgs","repos_url":"https://api.github.com/users/rdblue/repos","events_url":"https://api.github.com/users/rdblue/events{/privacy}","received_events_url":"https://api.github.com/users/rdblue/received_events","type":"User","site_admin":false},"body":"Not, not really. Those 10 reducers will run through multiple tasks. With 15 tasks per partition, each one will get at least one task. Each task will create a writer, write the data, and close that writer, creating one file in one partition (because we know all the data is for the same partition).","created_at":"2015-06-04T22:12:42Z","updated_at":"2015-06-04T22:12:42Z","html_url":"https://github.com/kite-sdk/kite/pull/377#discussion_r31771302","pull_request_url":"https://api.github.com/repos/kite-sdk/kite/pulls/377","_links":{"self":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/comments/31771302"},"html":{"href":"https://github.com/kite-sdk/kite/pull/377#discussion_r31771302"},"pull_request":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/377"}}},"pull_request":{"url":"https://api.github.com/repos/kite-sdk/kite/pulls/377","id":36938465,"html_url":"https://github.com/kite-sdk/kite/pull/377","diff_url":"https://github.com/kite-sdk/kite/pull/377.diff","patch_url":"https://github.com/kite-sdk/kite/pull/377.patch","issue_url":"https://api.github.com/repos/kite-sdk/kite/issues/377","number":377,"state":"open","locked":false,"title":"CDK-1011: Support a configurable number of writers per partition when…","user":{"login":"mkwhitacre","id":533989,"avatar_url":"https://avatars.githubusercontent.com/u/533989?v=3","gravatar_id":"","url":"https://api.github.com/users/mkwhitacre","html_url":"https://github.com/mkwhitacre","followers_url":"https://api.github.com/users/mkwhitacre/followers","following_url":"https://api.github.com/users/mkwhitacre/following{/other_user}","gists_url":"https://api.github.com/users/mkwhitacre/gists{/gist_id}","starred_url":"https://api.github.com/users/mkwhitacre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkwhitacre/subscriptions","organizations_url":"https://api.github.com/users/mkwhitacre/orgs","repos_url":"https://api.github.com/users/mkwhitacre/repos","events_url":"https://api.github.com/users/mkwhitacre/events{/privacy}","received_events_url":"https://api.github.com/users/mkwhitacre/received_events","type":"User","site_admin":false},"body":"… writing to a dataset, along with copying and compaction.\r\n\r\nI didn't see any current tests for the CrunchDatasets.partition(...) method or CompactionCommand/Task.  I can add some if necessary.","created_at":"2015-06-04T14:17:31Z","updated_at":"2015-06-04T22:12:42Z","closed_at":null,"merged_at":null,"merge_commit_sha":"72094613db5a1f7c23ac9cf8884da4e1479eb5ec","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/kite-sdk/kite/pulls/377/commits","review_comments_url":"https://api.github.com/repos/kite-sdk/kite/pulls/377/comments","review_comment_url":"https://api.github.com/repos/kite-sdk/kite/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kite-sdk/kite/issues/377/comments","statuses_url":"https://api.github.com/repos/kite-sdk/kite/statuses/3fd7ab854fe6f7f155c6993a2a2a129f7188e0ed","head":{"label":"mkwhitacre:CDK-1011","ref":"CDK-1011","sha":"3fd7ab854fe6f7f155c6993a2a2a129f7188e0ed","user":{"login":"mkwhitacre","id":533989,"avatar_url":"https://avatars.githubusercontent.com/u/533989?v=3","gravatar_id":"","url":"https://api.github.com/users/mkwhitacre","html_url":"https://github.com/mkwhitacre","followers_url":"https://api.github.com/users/mkwhitacre/followers","following_url":"https://api.github.com/users/mkwhitacre/following{/other_user}","gists_url":"https://api.github.com/users/mkwhitacre/gists{/gist_id}","starred_url":"https://api.github.com/users/mkwhitacre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkwhitacre/subscriptions","organizations_url":"https://api.github.com/users/mkwhitacre/orgs","repos_url":"https://api.github.com/users/mkwhitacre/repos","events_url":"https://api.github.com/users/mkwhitacre/events{/privacy}","received_events_url":"https://api.github.com/users/mkwhitacre/received_events","type":"User","site_admin":false},"repo":{"id":16356016,"name":"kite","full_name":"mkwhitacre/kite","owner":{"login":"mkwhitacre","id":533989,"avatar_url":"https://avatars.githubusercontent.com/u/533989?v=3","gravatar_id":"","url":"https://api.github.com/users/mkwhitacre","html_url":"https://github.com/mkwhitacre","followers_url":"https://api.github.com/users/mkwhitacre/followers","following_url":"https://api.github.com/users/mkwhitacre/following{/other_user}","gists_url":"https://api.github.com/users/mkwhitacre/gists{/gist_id}","starred_url":"https://api.github.com/users/mkwhitacre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkwhitacre/subscriptions","organizations_url":"https://api.github.com/users/mkwhitacre/orgs","repos_url":"https://api.github.com/users/mkwhitacre/repos","events_url":"https://api.github.com/users/mkwhitacre/events{/privacy}","received_events_url":"https://api.github.com/users/mkwhitacre/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mkwhitacre/kite","description":"Kite SDK","fork":true,"url":"https://api.github.com/repos/mkwhitacre/kite","forks_url":"https://api.github.com/repos/mkwhitacre/kite/forks","keys_url":"https://api.github.com/repos/mkwhitacre/kite/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mkwhitacre/kite/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mkwhitacre/kite/teams","hooks_url":"https://api.github.com/repos/mkwhitacre/kite/hooks","issue_events_url":"https://api.github.com/repos/mkwhitacre/kite/issues/events{/number}","events_url":"https://api.github.com/repos/mkwhitacre/kite/events","assignees_url":"https://api.github.com/repos/mkwhitacre/kite/assignees{/user}","branches_url":"https://api.github.com/repos/mkwhitacre/kite/branches{/branch}","tags_url":"https://api.github.com/repos/mkwhitacre/kite/tags","blobs_url":"https://api.github.com/repos/mkwhitacre/kite/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mkwhitacre/kite/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mkwhitacre/kite/git/refs{/sha}","trees_url":"https://api.github.com/repos/mkwhitacre/kite/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mkwhitacre/kite/statuses/{sha}","languages_url":"https://api.github.com/repos/mkwhitacre/kite/languages","stargazers_url":"https://api.github.com/repos/mkwhitacre/kite/stargazers","contributors_url":"https://api.github.com/repos/mkwhitacre/kite/contributors","subscribers_url":"https://api.github.com/repos/mkwhitacre/kite/subscribers","subscription_url":"https://api.github.com/repos/mkwhitacre/kite/subscription","commits_url":"https://api.github.com/repos/mkwhitacre/kite/commits{/sha}","git_commits_url":"https://api.github.com/repos/mkwhitacre/kite/git/commits{/sha}","comments_url":"https://api.github.com/repos/mkwhitacre/kite/comments{/number}","issue_comment_url":"https://api.github.com/repos/mkwhitacre/kite/issues/comments{/number}","contents_url":"https://api.github.com/repos/mkwhitacre/kite/contents/{+path}","compare_url":"https://api.github.com/repos/mkwhitacre/kite/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mkwhitacre/kite/merges","archive_url":"https://api.github.com/repos/mkwhitacre/kite/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mkwhitacre/kite/downloads","issues_url":"https://api.github.com/repos/mkwhitacre/kite/issues{/number}","pulls_url":"https://api.github.com/repos/mkwhitacre/kite/pulls{/number}","milestones_url":"https://api.github.com/repos/mkwhitacre/kite/milestones{/number}","notifications_url":"https://api.github.com/repos/mkwhitacre/kite/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mkwhitacre/kite/labels{/name}","releases_url":"https://api.github.com/repos/mkwhitacre/kite/releases{/id}","created_at":"2014-01-29T19:36:34Z","updated_at":"2014-06-24T13:16:39Z","pushed_at":"2015-06-04T22:04:56Z","git_url":"git://github.com/mkwhitacre/kite.git","ssh_url":"git@github.com:mkwhitacre/kite.git","clone_url":"https://github.com/mkwhitacre/kite.git","svn_url":"https://github.com/mkwhitacre/kite","homepage":"http://kitesdk.org/docs/current/","size":32168,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kite-sdk:master","ref":"master","sha":"93f58266f32e0eb9b576ca6f0c9f175ea39cfd31","user":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"repo":{"id":14829187,"name":"kite","full_name":"kite-sdk/kite","owner":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kite-sdk/kite","description":"Kite SDK","fork":false,"url":"https://api.github.com/repos/kite-sdk/kite","forks_url":"https://api.github.com/repos/kite-sdk/kite/forks","keys_url":"https://api.github.com/repos/kite-sdk/kite/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kite-sdk/kite/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kite-sdk/kite/teams","hooks_url":"https://api.github.com/repos/kite-sdk/kite/hooks","issue_events_url":"https://api.github.com/repos/kite-sdk/kite/issues/events{/number}","events_url":"https://api.github.com/repos/kite-sdk/kite/events","assignees_url":"https://api.github.com/repos/kite-sdk/kite/assignees{/user}","branches_url":"https://api.github.com/repos/kite-sdk/kite/branches{/branch}","tags_url":"https://api.github.com/repos/kite-sdk/kite/tags","blobs_url":"https://api.github.com/repos/kite-sdk/kite/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kite-sdk/kite/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kite-sdk/kite/git/refs{/sha}","trees_url":"https://api.github.com/repos/kite-sdk/kite/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kite-sdk/kite/statuses/{sha}","languages_url":"https://api.github.com/repos/kite-sdk/kite/languages","stargazers_url":"https://api.github.com/repos/kite-sdk/kite/stargazers","contributors_url":"https://api.github.com/repos/kite-sdk/kite/contributors","subscribers_url":"https://api.github.com/repos/kite-sdk/kite/subscribers","subscription_url":"https://api.github.com/repos/kite-sdk/kite/subscription","commits_url":"https://api.github.com/repos/kite-sdk/kite/commits{/sha}","git_commits_url":"https://api.github.com/repos/kite-sdk/kite/git/commits{/sha}","comments_url":"https://api.github.com/repos/kite-sdk/kite/comments{/number}","issue_comment_url":"https://api.github.com/repos/kite-sdk/kite/issues/comments{/number}","contents_url":"https://api.github.com/repos/kite-sdk/kite/contents/{+path}","compare_url":"https://api.github.com/repos/kite-sdk/kite/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kite-sdk/kite/merges","archive_url":"https://api.github.com/repos/kite-sdk/kite/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kite-sdk/kite/downloads","issues_url":"https://api.github.com/repos/kite-sdk/kite/issues{/number}","pulls_url":"https://api.github.com/repos/kite-sdk/kite/pulls{/number}","milestones_url":"https://api.github.com/repos/kite-sdk/kite/milestones{/number}","notifications_url":"https://api.github.com/repos/kite-sdk/kite/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kite-sdk/kite/labels{/name}","releases_url":"https://api.github.com/repos/kite-sdk/kite/releases{/id}","created_at":"2013-11-30T22:06:02Z","updated_at":"2015-06-02T15:03:28Z","pushed_at":"2015-06-04T22:08:15Z","git_url":"git://github.com/kite-sdk/kite.git","ssh_url":"git@github.com:kite-sdk/kite.git","clone_url":"https://github.com/kite-sdk/kite.git","svn_url":"https://github.com/kite-sdk/kite","homepage":"http://kitesdk.org/docs/current/","size":78081,"stargazers_count":132,"watchers_count":132,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":99,"mirror_url":null,"open_issues_count":15,"forks":99,"open_issues":15,"watchers":132,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/377"},"html":{"href":"https://github.com/kite-sdk/kite/pull/377"},"issue":{"href":"https://api.github.com/repos/kite-sdk/kite/issues/377"},"comments":{"href":"https://api.github.com/repos/kite-sdk/kite/issues/377/comments"},"review_comments":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/377/comments"},"review_comment":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/377/commits"},"statuses":{"href":"https://api.github.com/repos/kite-sdk/kite/statuses/3fd7ab854fe6f7f155c6993a2a2a129f7188e0ed"}}}}