{"action":"created","comment":{"url":"https://api.github.com/repos/druid-io/druid/pulls/comments/43650642","id":43650642,"diff_hunk":"@@ -342,9 +341,13 @@ public String getType()\n     @Override\n     public TaskStatus run(TaskToolbox toolbox) throws Exception\n     {\n-      log.info(\"Subs are good!  Italian BMT and Meatball are probably my favorite.\");\n-      convertSegment(toolbox, segment, indexSpec, force, validate);\n-      return success();\n+      if(toolbox.getTaskActionClient().submit(new LockUpgradeAction(getInterval()))) {\n+        log.info(\"Subs are good!  Italian BMT and Meatball are probably my favorite.\");\n+        convertSegment(toolbox, segment, indexSpec, force, validate);","path":"indexing-service/src/main/java/io/druid/indexing/common/task/ConvertSegmentTask.java","position":33,"original_position":33,"commit_id":"05671c1c9c8d9716567f9d0fa8d813a9609d9ae0","original_commit_id":"05671c1c9c8d9716567f9d0fa8d813a9609d9ae0","user":{"login":"gianm","id":1214075,"avatar_url":"https://avatars.githubusercontent.com/u/1214075?v=3","gravatar_id":"","url":"https://api.github.com/users/gianm","html_url":"https://github.com/gianm","followers_url":"https://api.github.com/users/gianm/followers","following_url":"https://api.github.com/users/gianm/following{/other_user}","gists_url":"https://api.github.com/users/gianm/gists{/gist_id}","starred_url":"https://api.github.com/users/gianm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gianm/subscriptions","organizations_url":"https://api.github.com/users/gianm/orgs","repos_url":"https://api.github.com/users/gianm/repos","events_url":"https://api.github.com/users/gianm/events{/privacy}","received_events_url":"https://api.github.com/users/gianm/received_events","type":"User","site_admin":false},"body":"this is actually not its own task; the upgrade should be hoisted up to the real task. (it just runs a loop over these things)\r\n\r\nAnd because of that, this is also a lot of work to be doing in a non-preemptible way, and maybe should be chunked (see comment on the archive task).","created_at":"2015-11-02T16:53:05Z","updated_at":"2015-11-02T16:53:05Z","html_url":"https://github.com/druid-io/druid/pull/1679#discussion_r43650642","pull_request_url":"https://api.github.com/repos/druid-io/druid/pulls/1679","_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments/43650642"},"html":{"href":"https://github.com/druid-io/druid/pull/1679#discussion_r43650642"},"pull_request":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1679"}}},"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/1679","id":43545999,"html_url":"https://github.com/druid-io/druid/pull/1679","diff_url":"https://github.com/druid-io/druid/pull/1679.diff","patch_url":"https://github.com/druid-io/druid/pull/1679.patch","issue_url":"https://api.github.com/repos/druid-io/druid/issues/1679","number":1679,"state":"open","locked":false,"title":"Priority based task locking","user":{"login":"pjain1","id":11787983,"avatar_url":"https://avatars.githubusercontent.com/u/11787983?v=3","gravatar_id":"","url":"https://api.github.com/users/pjain1","html_url":"https://github.com/pjain1","followers_url":"https://api.github.com/users/pjain1/followers","following_url":"https://api.github.com/users/pjain1/following{/other_user}","gists_url":"https://api.github.com/users/pjain1/gists{/gist_id}","starred_url":"https://api.github.com/users/pjain1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pjain1/subscriptions","organizations_url":"https://api.github.com/users/pjain1/orgs","repos_url":"https://api.github.com/users/pjain1/repos","events_url":"https://api.github.com/users/pjain1/events{/privacy}","received_events_url":"https://api.github.com/users/pjain1/received_events","type":"User","site_admin":false},"body":"This PR corresponds to the issue https://github.com/druid-io/druid/issues/1513\r\n\r\nDesign details -\r\n- The task priority is used for acquiring a lock on an interval for a datasource. Tasks with higher priority can preempt lower-priority tasks for the same datasource and interval if ran concurrently.\r\n  The flow for acquiring and upgrading Locks by a Task during TaskLifeCycle would be like -\r\n  - Check for TaskLocks on same datasource and overlapping interval\r\n    - If the one or more TaskLocks have same or higher priority then stop and retry after some time\r\n    - If no TaskLock is present or all the TaskLocks are of lower priority then check if any lock is an exclusive lock\r\n      - If no then revoke all the lower priority TaskLocks and create a new TaskLock\r\n      - If yes then stop and retry after some time\r\n  - Before publishing the segments, upgrade the TaskLock to exclusiveLock\r\n    - If upgrade successful publish the segment and return success status for the Task\r\n    - If no then return failure status for the Task\r\n\r\nLegacy tasks or tasks with \"0\" priority will have the respective default priorities as per the task type\r\n- Default priorities for task\r\n  - Realtime Index Task - 100\r\n  - Hadoop/Index Task - 75\r\n  - Merge/Append Task - 50\r\n  - Other Tasks - 25\r\n\r\n- Higher the number, higher the priority. \r\n\r\nFor example, if a Hadoop Index task is running and a Realtime Index task starts that wants to publish a segment for the same (or overlapping) interval for the same datasource, then it will override the task locks of the Hadoop Index task. Consequently, the Hadoop Index task will fail before publishing the segment.\r\n\r\nNote - There is no need to set this property, task automatically gets a default priority as per its type. However, if one wants to override the default priority it can be done by setting `priority` inside `context` property like this -\r\n```\r\n  \"context\" {\r\n    \"priority\" : \"80\"\r\n  }\r\n```\r\n\r\nMajor Implementation details -\r\n- Two new fields have been added to TaskLock namely priority and preemptive. \r\n- Modified and refactored tryLock method in TaskLockbox\r\n- Added upgradeLock method in TaskLockbox, MetaStorageActionHandler, SQLMetaStorageActionHandler, TaskStorage, MetadataTaskStorage and HeapMemoryTaskStorage\r\n- New Task Action named LockUpgradeAction has been added\r\n- Check for preemption of locks has been added in TaskActionToolbox which is performed before publishing the segments in SegmentInsertAction\r\n- Changed Task implementations to handle priority, preemption and check for lock validity\r\n- Modified ThreadPoolTaskRunner a bit to be helpful in testing lock override\r\n- Modified TaskSerdeTest to check for priority\r\n- Modified Task related docs\r\n- Added unit tests for TaskLock overriding in TaskLifeCylceTest\r\n- Added a task name TestIndexTask useful for unit testing\r\n\r\nPossible Future Enhancements - Proactively shutdown the tasks instead of waiting for them to fail eventually since their TaskLock has been revoked.\r\n","created_at":"2015-08-27T17:48:25Z","updated_at":"2015-11-02T16:53:05Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ad84e325d1f81b10b3bee6099a07bd00548246b9","assignee":null,"milestone":{"url":"https://api.github.com/repos/druid-io/druid/milestones/14","html_url":"https://github.com/druid-io/druid/milestones/0.9.0","labels_url":"https://api.github.com/repos/druid-io/druid/milestones/14/labels","id":1092727,"number":14,"title":"0.9.0","description":"","creator":{"login":"xvrl","id":815147,"avatar_url":"https://avatars.githubusercontent.com/u/815147?v=3","gravatar_id":"","url":"https://api.github.com/users/xvrl","html_url":"https://github.com/xvrl","followers_url":"https://api.github.com/users/xvrl/followers","following_url":"https://api.github.com/users/xvrl/following{/other_user}","gists_url":"https://api.github.com/users/xvrl/gists{/gist_id}","starred_url":"https://api.github.com/users/xvrl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xvrl/subscriptions","organizations_url":"https://api.github.com/users/xvrl/orgs","repos_url":"https://api.github.com/users/xvrl/repos","events_url":"https://api.github.com/users/xvrl/events{/privacy}","received_events_url":"https://api.github.com/users/xvrl/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":9,"state":"open","created_at":"2015-05-01T16:53:57Z","updated_at":"2015-10-30T23:01:03Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/druid-io/druid/pulls/1679/commits","review_comments_url":"https://api.github.com/repos/druid-io/druid/pulls/1679/comments","review_comment_url":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/1679/comments","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/05671c1c9c8d9716567f9d0fa8d813a9609d9ae0","head":{"label":"pjain1:priority_locking","ref":"priority_locking","sha":"05671c1c9c8d9716567f9d0fa8d813a9609d9ae0","user":{"login":"pjain1","id":11787983,"avatar_url":"https://avatars.githubusercontent.com/u/11787983?v=3","gravatar_id":"","url":"https://api.github.com/users/pjain1","html_url":"https://github.com/pjain1","followers_url":"https://api.github.com/users/pjain1/followers","following_url":"https://api.github.com/users/pjain1/following{/other_user}","gists_url":"https://api.github.com/users/pjain1/gists{/gist_id}","starred_url":"https://api.github.com/users/pjain1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pjain1/subscriptions","organizations_url":"https://api.github.com/users/pjain1/orgs","repos_url":"https://api.github.com/users/pjain1/repos","events_url":"https://api.github.com/users/pjain1/events{/privacy}","received_events_url":"https://api.github.com/users/pjain1/received_events","type":"User","site_admin":false},"repo":{"id":33489992,"name":"druid","full_name":"pjain1/druid","owner":{"login":"pjain1","id":11787983,"avatar_url":"https://avatars.githubusercontent.com/u/11787983?v=3","gravatar_id":"","url":"https://api.github.com/users/pjain1","html_url":"https://github.com/pjain1","followers_url":"https://api.github.com/users/pjain1/followers","following_url":"https://api.github.com/users/pjain1/following{/other_user}","gists_url":"https://api.github.com/users/pjain1/gists{/gist_id}","starred_url":"https://api.github.com/users/pjain1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pjain1/subscriptions","organizations_url":"https://api.github.com/users/pjain1/orgs","repos_url":"https://api.github.com/users/pjain1/repos","events_url":"https://api.github.com/users/pjain1/events{/privacy}","received_events_url":"https://api.github.com/users/pjain1/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/pjain1/druid","description":"Real²time Exploratory Analytics on Large Datasets","fork":true,"url":"https://api.github.com/repos/pjain1/druid","forks_url":"https://api.github.com/repos/pjain1/druid/forks","keys_url":"https://api.github.com/repos/pjain1/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pjain1/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pjain1/druid/teams","hooks_url":"https://api.github.com/repos/pjain1/druid/hooks","issue_events_url":"https://api.github.com/repos/pjain1/druid/issues/events{/number}","events_url":"https://api.github.com/repos/pjain1/druid/events","assignees_url":"https://api.github.com/repos/pjain1/druid/assignees{/user}","branches_url":"https://api.github.com/repos/pjain1/druid/branches{/branch}","tags_url":"https://api.github.com/repos/pjain1/druid/tags","blobs_url":"https://api.github.com/repos/pjain1/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pjain1/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pjain1/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/pjain1/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pjain1/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/pjain1/druid/languages","stargazers_url":"https://api.github.com/repos/pjain1/druid/stargazers","contributors_url":"https://api.github.com/repos/pjain1/druid/contributors","subscribers_url":"https://api.github.com/repos/pjain1/druid/subscribers","subscription_url":"https://api.github.com/repos/pjain1/druid/subscription","commits_url":"https://api.github.com/repos/pjain1/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/pjain1/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/pjain1/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/pjain1/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/pjain1/druid/contents/{+path}","compare_url":"https://api.github.com/repos/pjain1/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pjain1/druid/merges","archive_url":"https://api.github.com/repos/pjain1/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pjain1/druid/downloads","issues_url":"https://api.github.com/repos/pjain1/druid/issues{/number}","pulls_url":"https://api.github.com/repos/pjain1/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/pjain1/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/pjain1/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pjain1/druid/labels{/name}","releases_url":"https://api.github.com/repos/pjain1/druid/releases{/id}","created_at":"2015-04-06T15:44:13Z","updated_at":"2015-05-06T21:31:13Z","pushed_at":"2015-10-30T14:18:06Z","git_url":"git://github.com/pjain1/druid.git","ssh_url":"git@github.com:pjain1/druid.git","clone_url":"https://github.com/pjain1/druid.git","svn_url":"https://github.com/pjain1/druid","homepage":"http://druid.io","size":98970,"stargazers_count":1,"watchers_count":1,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"druid-io:master","ref":"master","sha":"56e0709da9d2a80a7be4de74c22950141fc5d439","user":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"repo":{"id":6358188,"name":"druid","full_name":"druid-io/druid","owner":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/druid-io/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":false,"url":"https://api.github.com/repos/druid-io/druid","forks_url":"https://api.github.com/repos/druid-io/druid/forks","keys_url":"https://api.github.com/repos/druid-io/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/druid-io/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/druid-io/druid/teams","hooks_url":"https://api.github.com/repos/druid-io/druid/hooks","issue_events_url":"https://api.github.com/repos/druid-io/druid/issues/events{/number}","events_url":"https://api.github.com/repos/druid-io/druid/events","assignees_url":"https://api.github.com/repos/druid-io/druid/assignees{/user}","branches_url":"https://api.github.com/repos/druid-io/druid/branches{/branch}","tags_url":"https://api.github.com/repos/druid-io/druid/tags","blobs_url":"https://api.github.com/repos/druid-io/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/druid-io/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/druid-io/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/druid-io/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/druid-io/druid/languages","stargazers_url":"https://api.github.com/repos/druid-io/druid/stargazers","contributors_url":"https://api.github.com/repos/druid-io/druid/contributors","subscribers_url":"https://api.github.com/repos/druid-io/druid/subscribers","subscription_url":"https://api.github.com/repos/druid-io/druid/subscription","commits_url":"https://api.github.com/repos/druid-io/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/druid-io/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/druid-io/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/druid-io/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/druid-io/druid/contents/{+path}","compare_url":"https://api.github.com/repos/druid-io/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/druid-io/druid/merges","archive_url":"https://api.github.com/repos/druid-io/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/druid-io/druid/downloads","issues_url":"https://api.github.com/repos/druid-io/druid/issues{/number}","pulls_url":"https://api.github.com/repos/druid-io/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/druid-io/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/druid-io/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/druid-io/druid/labels{/name}","releases_url":"https://api.github.com/repos/druid-io/druid/releases{/id}","created_at":"2012-10-23T19:08:07Z","updated_at":"2015-11-02T14:35:46Z","pushed_at":"2015-11-02T08:22:45Z","git_url":"git://github.com/druid-io/druid.git","ssh_url":"git@github.com:druid-io/druid.git","clone_url":"https://github.com/druid-io/druid.git","svn_url":"https://github.com/druid-io/druid","homepage":"http://druid.io","size":183455,"stargazers_count":2125,"watchers_count":2125,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":482,"mirror_url":null,"open_issues_count":213,"forks":482,"open_issues":213,"watchers":2125,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1679"},"html":{"href":"https://github.com/druid-io/druid/pull/1679"},"issue":{"href":"https://api.github.com/repos/druid-io/druid/issues/1679"},"comments":{"href":"https://api.github.com/repos/druid-io/druid/issues/1679/comments"},"review_comments":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1679/comments"},"review_comment":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1679/commits"},"statuses":{"href":"https://api.github.com/repos/druid-io/druid/statuses/05671c1c9c8d9716567f9d0fa8d813a9609d9ae0"}}}}