{"action":"created","comment":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/comments/39703446","id":39703446,"diff_hunk":"@@ -46,16 +50,46 @@\n import gobblin.util.DatasetFilterUtils;\n import gobblin.util.HadoopUtils;\n \n+import static gobblin.compaction.mapreduce.MRCompactor.FolderState.*;\n+import static gobblin.compaction.mapreduce.MRCompactorJobRunner.Status.*;\n+\n \n /**\n  * MapReduce-based {@link gobblin.compaction.Compactor}.\n  *\n  * @author ziliu\n  */\n+\n public class MRCompactor implements Compactor {\n \n   private static final Logger LOG = LoggerFactory.getLogger(MRCompactor.class);\n-  private static final List<Job> RUNNING_MR_JOBS = Lists.newCopyOnWriteArrayList();\n+\n+  private static final double HIGH_PRIORITY = 3.0;\n+  private static final double NORMAL_PRIORITY = 2.0;\n+  private static final double LOW_PRIORITY = 1.0;\n+  private static final double PRIORITY_REDUCTION_FACTOR = 1.0 / 3.0;\n+  private static final long COMPACTION_JOB_WAIT_INTERVAL_SECONDS = 10;\n+  private static final Map<String, Job> RUNNING_MR_JOBS = Maps.newConcurrentMap();","path":"gobblin-compaction/src/main/java/gobblin/compaction/mapreduce/MRCompactor.java","position":61,"original_position":61,"commit_id":"963e782af1f921787d099873bbe88ab9f5ba6398","original_commit_id":"963e782af1f921787d099873bbe88ab9f5ba6398","user":{"login":"zliu41","id":6342538,"avatar_url":"https://avatars.githubusercontent.com/u/6342538?v=3","gravatar_id":"","url":"https://api.github.com/users/zliu41","html_url":"https://github.com/zliu41","followers_url":"https://api.github.com/users/zliu41/followers","following_url":"https://api.github.com/users/zliu41/following{/other_user}","gists_url":"https://api.github.com/users/zliu41/gists{/gist_id}","starred_url":"https://api.github.com/users/zliu41/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zliu41/subscriptions","organizations_url":"https://api.github.com/users/zliu41/orgs","repos_url":"https://api.github.com/users/zliu41/repos","events_url":"https://api.github.com/users/zliu41/events{/privacy}","received_events_url":"https://api.github.com/users/zliu41/received_events","type":"User","site_admin":false},"body":"Then I'll need to pass an instance of MRCompactor to `MRCompactorJobRunner` which creates the Hadoop job.","created_at":"2015-09-17T01:12:52Z","updated_at":"2015-09-17T01:12:52Z","html_url":"https://github.com/linkedin/gobblin/pull/329#discussion_r39703446","pull_request_url":"https://api.github.com/repos/linkedin/gobblin/pulls/329","_links":{"self":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/comments/39703446"},"html":{"href":"https://github.com/linkedin/gobblin/pull/329#discussion_r39703446"},"pull_request":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/329"}}},"pull_request":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/329","id":45107395,"html_url":"https://github.com/linkedin/gobblin/pull/329","diff_url":"https://github.com/linkedin/gobblin/pull/329.diff","patch_url":"https://github.com/linkedin/gobblin/pull/329.patch","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/329","number":329,"state":"open","locked":false,"title":"Hourly Compaction","user":{"login":"zliu41","id":6342538,"avatar_url":"https://avatars.githubusercontent.com/u/6342538?v=3","gravatar_id":"","url":"https://api.github.com/users/zliu41","html_url":"https://github.com/zliu41","followers_url":"https://api.github.com/users/zliu41/followers","following_url":"https://api.github.com/users/zliu41/following{/other_user}","gists_url":"https://api.github.com/users/zliu41/gists{/gist_id}","starred_url":"https://api.github.com/users/zliu41/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zliu41/subscriptions","organizations_url":"https://api.github.com/users/zliu41/orgs","repos_url":"https://api.github.com/users/zliu41/repos","events_url":"https://api.github.com/users/zliu41/events{/privacy}","received_events_url":"https://api.github.com/users/zliu41/received_events","type":"User","site_admin":false},"body":"How this works:\r\n\r\n- There is a `FolderState` associated with each input folder. Possible values are `YES`, `NO`, `PENDING`, `COMPLETE`. Initially the status of all input folders is `NO`.\r\n- When compaction starts, it submits data completeness verification requests in a thread pool, one request per input folder, in the order of their priorities (high priority = 3.0, normal = 2.0, low = 1.0). After data completeness verification for a folder starts, the folder status changes to `PENDING`.\r\n- Meanwhile, a job submitter submits compaction MR jobs for the input folders to another thread pool. The compaction MR job of an input folder is only submitted if the folder status is `PENDING` or `YES`, and there is not already a compaction MR job running for this input folder. If the status is `NO`, there's no point to run the compaction MR job since we don't know if the data is complete or not. If the status is `COMPLETE`, it means compaction for this input folder has finished (which may have either succeeded or failed), and the job submitter will no longer submit MR jobs for this input folder. If there's already a compaction MR job running for this input folder, the job submitter will not submit it again until the current run finishes.\r\n - If, when a compaction finishes, the status of the input folder is still `PENDING`, it will not publish the data and will discard the work it has done.\r\n- Once the data completeness verification for a folder finishes:\r\n - if the data is complete, the status of the folder is changed to `YES`, and the thread which is running the compaction job for this folder (if exists) is notified, which will proceed to publish the data. \r\n - if the data is incomplete, the status of the folder is changed to `NO`, and the thread which is running the compaction job for this folder will be asked to terminate asap without publishing the data. Another data completeness verification for this folder will be submitted again with priority reduced by 1/3.\r\n- Once the compaction MR job for a folder finishes, if the job successfully published the data, or failed for reasons other than data incompleteness, the status of the input folder will be changed to COMPLETE, and the job submitter will no longer submit jobs for this input folder. Otherwise, the job submitter will attempt to re-submit the compaction MR job for this input folder with priority reduced by 1/3 (it will not re-submit the job until the input folder status is changed to PENDING or YES, as mentioned before).\r\n\r\n","created_at":"2015-09-15T21:52:54Z","updated_at":"2015-09-17T01:12:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bf604bbf47aa7729960aaa41334e4093ebff9826","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/linkedin/gobblin/pulls/329/commits","review_comments_url":"https://api.github.com/repos/linkedin/gobblin/pulls/329/comments","review_comment_url":"https://api.github.com/repos/linkedin/gobblin/pulls/comments{/number}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/329/comments","statuses_url":"https://api.github.com/repos/linkedin/gobblin/statuses/963e782af1f921787d099873bbe88ab9f5ba6398","head":{"label":"zliu41:hc","ref":"hc","sha":"963e782af1f921787d099873bbe88ab9f5ba6398","user":{"login":"zliu41","id":6342538,"avatar_url":"https://avatars.githubusercontent.com/u/6342538?v=3","gravatar_id":"","url":"https://api.github.com/users/zliu41","html_url":"https://github.com/zliu41","followers_url":"https://api.github.com/users/zliu41/followers","following_url":"https://api.github.com/users/zliu41/following{/other_user}","gists_url":"https://api.github.com/users/zliu41/gists{/gist_id}","starred_url":"https://api.github.com/users/zliu41/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zliu41/subscriptions","organizations_url":"https://api.github.com/users/zliu41/orgs","repos_url":"https://api.github.com/users/zliu41/repos","events_url":"https://api.github.com/users/zliu41/events{/privacy}","received_events_url":"https://api.github.com/users/zliu41/received_events","type":"User","site_admin":false},"repo":{"id":32048123,"name":"gobblin","full_name":"zliu41/gobblin","owner":{"login":"zliu41","id":6342538,"avatar_url":"https://avatars.githubusercontent.com/u/6342538?v=3","gravatar_id":"","url":"https://api.github.com/users/zliu41","html_url":"https://github.com/zliu41","followers_url":"https://api.github.com/users/zliu41/followers","following_url":"https://api.github.com/users/zliu41/following{/other_user}","gists_url":"https://api.github.com/users/zliu41/gists{/gist_id}","starred_url":"https://api.github.com/users/zliu41/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zliu41/subscriptions","organizations_url":"https://api.github.com/users/zliu41/orgs","repos_url":"https://api.github.com/users/zliu41/repos","events_url":"https://api.github.com/users/zliu41/events{/privacy}","received_events_url":"https://api.github.com/users/zliu41/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/zliu41/gobblin","description":"","fork":true,"url":"https://api.github.com/repos/zliu41/gobblin","forks_url":"https://api.github.com/repos/zliu41/gobblin/forks","keys_url":"https://api.github.com/repos/zliu41/gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/zliu41/gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/zliu41/gobblin/teams","hooks_url":"https://api.github.com/repos/zliu41/gobblin/hooks","issue_events_url":"https://api.github.com/repos/zliu41/gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/zliu41/gobblin/events","assignees_url":"https://api.github.com/repos/zliu41/gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/zliu41/gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/zliu41/gobblin/tags","blobs_url":"https://api.github.com/repos/zliu41/gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/zliu41/gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/zliu41/gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/zliu41/gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/zliu41/gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/zliu41/gobblin/languages","stargazers_url":"https://api.github.com/repos/zliu41/gobblin/stargazers","contributors_url":"https://api.github.com/repos/zliu41/gobblin/contributors","subscribers_url":"https://api.github.com/repos/zliu41/gobblin/subscribers","subscription_url":"https://api.github.com/repos/zliu41/gobblin/subscription","commits_url":"https://api.github.com/repos/zliu41/gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/zliu41/gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/zliu41/gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/zliu41/gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/zliu41/gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/zliu41/gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/zliu41/gobblin/merges","archive_url":"https://api.github.com/repos/zliu41/gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/zliu41/gobblin/downloads","issues_url":"https://api.github.com/repos/zliu41/gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/zliu41/gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/zliu41/gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/zliu41/gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/zliu41/gobblin/labels{/name}","releases_url":"https://api.github.com/repos/zliu41/gobblin/releases{/id}","created_at":"2015-03-12T00:17:48Z","updated_at":"2015-05-07T16:27:07Z","pushed_at":"2015-09-16T16:49:49Z","git_url":"git://github.com/zliu41/gobblin.git","ssh_url":"git@github.com:zliu41/gobblin.git","clone_url":"https://github.com/zliu41/gobblin.git","svn_url":"https://github.com/zliu41/gobblin","homepage":null,"size":79064,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"linkedin:master","ref":"master","sha":"2e076bcdde5fb893675f105eca9a9345bfdc112f","user":{"login":"linkedin","id":357098,"avatar_url":"https://avatars.githubusercontent.com/u/357098?v=3","gravatar_id":"","url":"https://api.github.com/users/linkedin","html_url":"https://github.com/linkedin","followers_url":"https://api.github.com/users/linkedin/followers","following_url":"https://api.github.com/users/linkedin/following{/other_user}","gists_url":"https://api.github.com/users/linkedin/gists{/gist_id}","starred_url":"https://api.github.com/users/linkedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linkedin/subscriptions","organizations_url":"https://api.github.com/users/linkedin/orgs","repos_url":"https://api.github.com/users/linkedin/repos","events_url":"https://api.github.com/users/linkedin/events{/privacy}","received_events_url":"https://api.github.com/users/linkedin/received_events","type":"Organization","site_admin":false},"repo":{"id":27392543,"name":"gobblin","full_name":"linkedin/gobblin","owner":{"login":"linkedin","id":357098,"avatar_url":"https://avatars.githubusercontent.com/u/357098?v=3","gravatar_id":"","url":"https://api.github.com/users/linkedin","html_url":"https://github.com/linkedin","followers_url":"https://api.github.com/users/linkedin/followers","following_url":"https://api.github.com/users/linkedin/following{/other_user}","gists_url":"https://api.github.com/users/linkedin/gists{/gist_id}","starred_url":"https://api.github.com/users/linkedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linkedin/subscriptions","organizations_url":"https://api.github.com/users/linkedin/orgs","repos_url":"https://api.github.com/users/linkedin/repos","events_url":"https://api.github.com/users/linkedin/events{/privacy}","received_events_url":"https://api.github.com/users/linkedin/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/linkedin/gobblin","description":"Universal data ingestion framework for Hadoop.","fork":false,"url":"https://api.github.com/repos/linkedin/gobblin","forks_url":"https://api.github.com/repos/linkedin/gobblin/forks","keys_url":"https://api.github.com/repos/linkedin/gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/linkedin/gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/linkedin/gobblin/teams","hooks_url":"https://api.github.com/repos/linkedin/gobblin/hooks","issue_events_url":"https://api.github.com/repos/linkedin/gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/linkedin/gobblin/events","assignees_url":"https://api.github.com/repos/linkedin/gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/linkedin/gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/linkedin/gobblin/tags","blobs_url":"https://api.github.com/repos/linkedin/gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/linkedin/gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/linkedin/gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/linkedin/gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/linkedin/gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/linkedin/gobblin/languages","stargazers_url":"https://api.github.com/repos/linkedin/gobblin/stargazers","contributors_url":"https://api.github.com/repos/linkedin/gobblin/contributors","subscribers_url":"https://api.github.com/repos/linkedin/gobblin/subscribers","subscription_url":"https://api.github.com/repos/linkedin/gobblin/subscription","commits_url":"https://api.github.com/repos/linkedin/gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/linkedin/gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/linkedin/gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/linkedin/gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/linkedin/gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/linkedin/gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/linkedin/gobblin/merges","archive_url":"https://api.github.com/repos/linkedin/gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/linkedin/gobblin/downloads","issues_url":"https://api.github.com/repos/linkedin/gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/linkedin/gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/linkedin/gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/linkedin/gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/linkedin/gobblin/labels{/name}","releases_url":"https://api.github.com/repos/linkedin/gobblin/releases{/id}","created_at":"2014-12-01T18:10:50Z","updated_at":"2015-09-16T21:00:50Z","pushed_at":"2015-09-16T23:48:09Z","git_url":"git://github.com/linkedin/gobblin.git","ssh_url":"git@github.com:linkedin/gobblin.git","clone_url":"https://github.com/linkedin/gobblin.git","svn_url":"https://github.com/linkedin/gobblin","homepage":"","size":159574,"stargazers_count":177,"watchers_count":177,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":60,"mirror_url":null,"open_issues_count":7,"forks":60,"open_issues":7,"watchers":177,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/329"},"html":{"href":"https://github.com/linkedin/gobblin/pull/329"},"issue":{"href":"https://api.github.com/repos/linkedin/gobblin/issues/329"},"comments":{"href":"https://api.github.com/repos/linkedin/gobblin/issues/329/comments"},"review_comments":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/329/comments"},"review_comment":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/329/commits"},"statuses":{"href":"https://api.github.com/repos/linkedin/gobblin/statuses/963e782af1f921787d099873bbe88ab9f5ba6398"}}}}