{"action":"created","comment":{"url":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/comments/39853761","id":39853761,"diff_hunk":"@@ -54,9 +54,23 @@\n     private final SparkJobContext _sparkJobContext;\n     private final JavaSparkContext _sparkContext;\n \n+    private final Integer _minPartitions;\n+\n     public SparkAnalysisRunner(JavaSparkContext sparkContext, SparkJobContext sparkJobContext) {\n+        this(sparkContext, sparkJobContext, null);\n+    }\n+\n+    public SparkAnalysisRunner(JavaSparkContext sparkContext, SparkJobContext sparkJobContext, Integer minPartitions) {\n         _sparkContext = sparkContext;\n         _sparkJobContext = sparkJobContext;\n+        if ((minPartitions != null) && (minPartitions > 0)) {\n+            _minPartitions = minPartitions;\n+        } else {\n+            logger.warn(","path":"engine/env/spark/src/main/java/org/datacleaner/spark/SparkAnalysisRunner.java","position":16,"original_position":16,"commit_id":"282ec3bb511bdf42314cbef064c7ff22b76b5109","original_commit_id":"282ec3bb511bdf42314cbef064c7ff22b76b5109","user":{"login":"kaspersorensen","id":291450,"avatar_url":"https://avatars.githubusercontent.com/u/291450?v=3","gravatar_id":"","url":"https://api.github.com/users/kaspersorensen","html_url":"https://github.com/kaspersorensen","followers_url":"https://api.github.com/users/kaspersorensen/followers","following_url":"https://api.github.com/users/kaspersorensen/following{/other_user}","gists_url":"https://api.github.com/users/kaspersorensen/gists{/gist_id}","starred_url":"https://api.github.com/users/kaspersorensen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaspersorensen/subscriptions","organizations_url":"https://api.github.com/users/kaspersorensen/orgs","repos_url":"https://api.github.com/users/kaspersorensen/repos","events_url":"https://api.github.com/users/kaspersorensen/events{/privacy}","received_events_url":"https://api.github.com/users/kaspersorensen/received_events","type":"User","site_admin":false},"body":"Here it's going to raise a warning for something that is the default behaviour ... I would rather only throw an exception if minPartitions is non-null and less-or-equal to 0.","created_at":"2015-09-18T13:25:14Z","updated_at":"2015-09-18T13:25:14Z","html_url":"https://github.com/datacleaner/DataCleaner/pull/674#discussion_r39853761","pull_request_url":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/674","_links":{"self":{"href":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/comments/39853761"},"html":{"href":"https://github.com/datacleaner/DataCleaner/pull/674#discussion_r39853761"},"pull_request":{"href":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/674"}}},"pull_request":{"url":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/674","id":45265555,"html_url":"https://github.com/datacleaner/DataCleaner/pull/674","diff_url":"https://github.com/datacleaner/DataCleaner/pull/674.diff","patch_url":"https://github.com/datacleaner/DataCleaner/pull/674.patch","issue_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/674","number":674,"state":"open","locked":false,"title":"Feature/647 value dist reducer non pairwise","user":{"login":"tomaszguzialek","id":1483194,"avatar_url":"https://avatars.githubusercontent.com/u/1483194?v=3","gravatar_id":"","url":"https://api.github.com/users/tomaszguzialek","html_url":"https://github.com/tomaszguzialek","followers_url":"https://api.github.com/users/tomaszguzialek/followers","following_url":"https://api.github.com/users/tomaszguzialek/following{/other_user}","gists_url":"https://api.github.com/users/tomaszguzialek/gists{/gist_id}","starred_url":"https://api.github.com/users/tomaszguzialek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomaszguzialek/subscriptions","organizations_url":"https://api.github.com/users/tomaszguzialek/orgs","repos_url":"https://api.github.com/users/tomaszguzialek/repos","events_url":"https://api.github.com/users/tomaszguzialek/events{/privacy}","received_events_url":"https://api.github.com/users/tomaszguzialek/received_events","type":"User","site_admin":false},"body":"This PR is competitive to #660, so when reviewing have a look at both approaches and merge only one. This approach is iterative reducing while #660 merges one partial result at a time.\r\n\r\nFixes #647 \r\n","created_at":"2015-09-17T08:48:10Z","updated_at":"2015-09-18T13:25:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"654370ad26e7236bf86c301c1ef9493293b9d12c","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/674/commits","review_comments_url":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/674/comments","review_comment_url":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/comments{/number}","comments_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/674/comments","statuses_url":"https://api.github.com/repos/datacleaner/DataCleaner/statuses/282ec3bb511bdf42314cbef064c7ff22b76b5109","head":{"label":"datacleaner:feature/647-value-dist-reducer-non-pairwise","ref":"feature/647-value-dist-reducer-non-pairwise","sha":"282ec3bb511bdf42314cbef064c7ff22b76b5109","user":{"login":"datacleaner","id":1665195,"avatar_url":"https://avatars.githubusercontent.com/u/1665195?v=3","gravatar_id":"","url":"https://api.github.com/users/datacleaner","html_url":"https://github.com/datacleaner","followers_url":"https://api.github.com/users/datacleaner/followers","following_url":"https://api.github.com/users/datacleaner/following{/other_user}","gists_url":"https://api.github.com/users/datacleaner/gists{/gist_id}","starred_url":"https://api.github.com/users/datacleaner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datacleaner/subscriptions","organizations_url":"https://api.github.com/users/datacleaner/orgs","repos_url":"https://api.github.com/users/datacleaner/repos","events_url":"https://api.github.com/users/datacleaner/events{/privacy}","received_events_url":"https://api.github.com/users/datacleaner/received_events","type":"Organization","site_admin":false},"repo":{"id":17034809,"name":"DataCleaner","full_name":"datacleaner/DataCleaner","owner":{"login":"datacleaner","id":1665195,"avatar_url":"https://avatars.githubusercontent.com/u/1665195?v=3","gravatar_id":"","url":"https://api.github.com/users/datacleaner","html_url":"https://github.com/datacleaner","followers_url":"https://api.github.com/users/datacleaner/followers","following_url":"https://api.github.com/users/datacleaner/following{/other_user}","gists_url":"https://api.github.com/users/datacleaner/gists{/gist_id}","starred_url":"https://api.github.com/users/datacleaner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datacleaner/subscriptions","organizations_url":"https://api.github.com/users/datacleaner/orgs","repos_url":"https://api.github.com/users/datacleaner/repos","events_url":"https://api.github.com/users/datacleaner/events{/privacy}","received_events_url":"https://api.github.com/users/datacleaner/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/datacleaner/DataCleaner","description":"The premier open source Data Quality solution","fork":false,"url":"https://api.github.com/repos/datacleaner/DataCleaner","forks_url":"https://api.github.com/repos/datacleaner/DataCleaner/forks","keys_url":"https://api.github.com/repos/datacleaner/DataCleaner/keys{/key_id}","collaborators_url":"https://api.github.com/repos/datacleaner/DataCleaner/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/datacleaner/DataCleaner/teams","hooks_url":"https://api.github.com/repos/datacleaner/DataCleaner/hooks","issue_events_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/events{/number}","events_url":"https://api.github.com/repos/datacleaner/DataCleaner/events","assignees_url":"https://api.github.com/repos/datacleaner/DataCleaner/assignees{/user}","branches_url":"https://api.github.com/repos/datacleaner/DataCleaner/branches{/branch}","tags_url":"https://api.github.com/repos/datacleaner/DataCleaner/tags","blobs_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/refs{/sha}","trees_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/trees{/sha}","statuses_url":"https://api.github.com/repos/datacleaner/DataCleaner/statuses/{sha}","languages_url":"https://api.github.com/repos/datacleaner/DataCleaner/languages","stargazers_url":"https://api.github.com/repos/datacleaner/DataCleaner/stargazers","contributors_url":"https://api.github.com/repos/datacleaner/DataCleaner/contributors","subscribers_url":"https://api.github.com/repos/datacleaner/DataCleaner/subscribers","subscription_url":"https://api.github.com/repos/datacleaner/DataCleaner/subscription","commits_url":"https://api.github.com/repos/datacleaner/DataCleaner/commits{/sha}","git_commits_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/commits{/sha}","comments_url":"https://api.github.com/repos/datacleaner/DataCleaner/comments{/number}","issue_comment_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/comments{/number}","contents_url":"https://api.github.com/repos/datacleaner/DataCleaner/contents/{+path}","compare_url":"https://api.github.com/repos/datacleaner/DataCleaner/compare/{base}...{head}","merges_url":"https://api.github.com/repos/datacleaner/DataCleaner/merges","archive_url":"https://api.github.com/repos/datacleaner/DataCleaner/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/datacleaner/DataCleaner/downloads","issues_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues{/number}","pulls_url":"https://api.github.com/repos/datacleaner/DataCleaner/pulls{/number}","milestones_url":"https://api.github.com/repos/datacleaner/DataCleaner/milestones{/number}","notifications_url":"https://api.github.com/repos/datacleaner/DataCleaner/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/datacleaner/DataCleaner/labels{/name}","releases_url":"https://api.github.com/repos/datacleaner/DataCleaner/releases{/id}","created_at":"2014-02-20T20:37:25Z","updated_at":"2015-09-18T11:22:39Z","pushed_at":"2015-09-18T12:46:27Z","git_url":"git://github.com/datacleaner/DataCleaner.git","ssh_url":"git@github.com:datacleaner/DataCleaner.git","clone_url":"https://github.com/datacleaner/DataCleaner.git","svn_url":"https://github.com/datacleaner/DataCleaner","homepage":null,"size":177283,"stargazers_count":35,"watchers_count":35,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":24,"mirror_url":null,"open_issues_count":137,"forks":24,"open_issues":137,"watchers":35,"default_branch":"master"}},"base":{"label":"datacleaner:master","ref":"master","sha":"a0a8030b58a1db149dc6700dae0512d59bc33c2c","user":{"login":"datacleaner","id":1665195,"avatar_url":"https://avatars.githubusercontent.com/u/1665195?v=3","gravatar_id":"","url":"https://api.github.com/users/datacleaner","html_url":"https://github.com/datacleaner","followers_url":"https://api.github.com/users/datacleaner/followers","following_url":"https://api.github.com/users/datacleaner/following{/other_user}","gists_url":"https://api.github.com/users/datacleaner/gists{/gist_id}","starred_url":"https://api.github.com/users/datacleaner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datacleaner/subscriptions","organizations_url":"https://api.github.com/users/datacleaner/orgs","repos_url":"https://api.github.com/users/datacleaner/repos","events_url":"https://api.github.com/users/datacleaner/events{/privacy}","received_events_url":"https://api.github.com/users/datacleaner/received_events","type":"Organization","site_admin":false},"repo":{"id":17034809,"name":"DataCleaner","full_name":"datacleaner/DataCleaner","owner":{"login":"datacleaner","id":1665195,"avatar_url":"https://avatars.githubusercontent.com/u/1665195?v=3","gravatar_id":"","url":"https://api.github.com/users/datacleaner","html_url":"https://github.com/datacleaner","followers_url":"https://api.github.com/users/datacleaner/followers","following_url":"https://api.github.com/users/datacleaner/following{/other_user}","gists_url":"https://api.github.com/users/datacleaner/gists{/gist_id}","starred_url":"https://api.github.com/users/datacleaner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datacleaner/subscriptions","organizations_url":"https://api.github.com/users/datacleaner/orgs","repos_url":"https://api.github.com/users/datacleaner/repos","events_url":"https://api.github.com/users/datacleaner/events{/privacy}","received_events_url":"https://api.github.com/users/datacleaner/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/datacleaner/DataCleaner","description":"The premier open source Data Quality solution","fork":false,"url":"https://api.github.com/repos/datacleaner/DataCleaner","forks_url":"https://api.github.com/repos/datacleaner/DataCleaner/forks","keys_url":"https://api.github.com/repos/datacleaner/DataCleaner/keys{/key_id}","collaborators_url":"https://api.github.com/repos/datacleaner/DataCleaner/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/datacleaner/DataCleaner/teams","hooks_url":"https://api.github.com/repos/datacleaner/DataCleaner/hooks","issue_events_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/events{/number}","events_url":"https://api.github.com/repos/datacleaner/DataCleaner/events","assignees_url":"https://api.github.com/repos/datacleaner/DataCleaner/assignees{/user}","branches_url":"https://api.github.com/repos/datacleaner/DataCleaner/branches{/branch}","tags_url":"https://api.github.com/repos/datacleaner/DataCleaner/tags","blobs_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/refs{/sha}","trees_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/trees{/sha}","statuses_url":"https://api.github.com/repos/datacleaner/DataCleaner/statuses/{sha}","languages_url":"https://api.github.com/repos/datacleaner/DataCleaner/languages","stargazers_url":"https://api.github.com/repos/datacleaner/DataCleaner/stargazers","contributors_url":"https://api.github.com/repos/datacleaner/DataCleaner/contributors","subscribers_url":"https://api.github.com/repos/datacleaner/DataCleaner/subscribers","subscription_url":"https://api.github.com/repos/datacleaner/DataCleaner/subscription","commits_url":"https://api.github.com/repos/datacleaner/DataCleaner/commits{/sha}","git_commits_url":"https://api.github.com/repos/datacleaner/DataCleaner/git/commits{/sha}","comments_url":"https://api.github.com/repos/datacleaner/DataCleaner/comments{/number}","issue_comment_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues/comments{/number}","contents_url":"https://api.github.com/repos/datacleaner/DataCleaner/contents/{+path}","compare_url":"https://api.github.com/repos/datacleaner/DataCleaner/compare/{base}...{head}","merges_url":"https://api.github.com/repos/datacleaner/DataCleaner/merges","archive_url":"https://api.github.com/repos/datacleaner/DataCleaner/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/datacleaner/DataCleaner/downloads","issues_url":"https://api.github.com/repos/datacleaner/DataCleaner/issues{/number}","pulls_url":"https://api.github.com/repos/datacleaner/DataCleaner/pulls{/number}","milestones_url":"https://api.github.com/repos/datacleaner/DataCleaner/milestones{/number}","notifications_url":"https://api.github.com/repos/datacleaner/DataCleaner/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/datacleaner/DataCleaner/labels{/name}","releases_url":"https://api.github.com/repos/datacleaner/DataCleaner/releases{/id}","created_at":"2014-02-20T20:37:25Z","updated_at":"2015-09-18T11:22:39Z","pushed_at":"2015-09-18T12:46:27Z","git_url":"git://github.com/datacleaner/DataCleaner.git","ssh_url":"git@github.com:datacleaner/DataCleaner.git","clone_url":"https://github.com/datacleaner/DataCleaner.git","svn_url":"https://github.com/datacleaner/DataCleaner","homepage":null,"size":177283,"stargazers_count":35,"watchers_count":35,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":24,"mirror_url":null,"open_issues_count":137,"forks":24,"open_issues":137,"watchers":35,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/674"},"html":{"href":"https://github.com/datacleaner/DataCleaner/pull/674"},"issue":{"href":"https://api.github.com/repos/datacleaner/DataCleaner/issues/674"},"comments":{"href":"https://api.github.com/repos/datacleaner/DataCleaner/issues/674/comments"},"review_comments":{"href":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/674/comments"},"review_comment":{"href":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/datacleaner/DataCleaner/pulls/674/commits"},"statuses":{"href":"https://api.github.com/repos/datacleaner/DataCleaner/statuses/282ec3bb511bdf42314cbef064c7ff22b76b5109"}}}}